[alexreinking]

Some short vectors of narrow integers are better computed in general-purpose registers than in SIMD registers: "SIMD within a register" (SWAR). A typical case is unpacking a few bytes with shifts and masks and handing the result to scalar code or to a few lanes of a vector. Doing that in Neon or SSE costs round-trips between the two register files and occupies the vector pipes. Hand-written kernels use SWAR for exactly this. For example, GGML decodes q4_K's twelve bytes of 6-bit scales and mins as three uint32_t words with shifts and masks.

Today Halide can't express this choice: vectorize always means SIMD instructions on SIMD registers. This PR separates the two halves of that choice:

• Compute, a property of the vectorized loop: vectorize(var, factor, VectorizeIn::SWAR) computes the loop's lanes with SWAR arithmetic on packed integer words.
• Storage, a property of the Func: store_in(MemoryType::GPR) keeps an allocation as integer words in general-purpose registers.

Either may be used without the other.

f.split(k, j, i, 4).unroll(j).vectorize(i, 4, VectorizeIn::SWAR);

SWAR is a loop type rather than a storage choice because it describes how a loop's vectors are computed, not where a Func lives. The same Func can have stages vectorized either way, and a SWAR loop's results can be handed straight to SIMD consumers. It also composes with SIMD vectorization. A SWAR loop may sit inside a SIMD loop, so that each SIMD lane holds a word of packed lanes ("SWAR within a vector register"). For example, this decodes four blocks' scales at once:

f.split(k, j, i, 4).unroll(j).vectorize(i, 4, VectorizeIn::SWAR).vectorize(b, 4);

The storage choice is needed separately because SWAR lowering is followed by simplification. If a stage only copies bytes (staging a block's index bytes for a table lookup, say), the simplifier folds it back to an ordinary vector load and store. LLVM then keeps the stage in a SIMD register and extracts each byte with a lane move. MemoryType::GPR makes codegen store each dense vector of the allocation as one integer, so LLVM keeps it in GPRs and the lookups extract their indices with shifts and masks. It does this whether the stage was vectorized as SIMD or SWAR.

idx.compute_at(g, b).store_in(MemoryType::GPR).split(k, ko, ki, 8).unroll(ko).vectorize(ki, 8);

What it does.

• enum class VectorizeIn { SIMD, SWAR } and Func/Stage::vectorize(var, [factor,] VectorizeIn, [tail]) overloads. VectorizeIn::SIMD is the existing behavior.
• ForType::VectorizedSWAR, with an is_vectorized() helper used at the sites that check for vectorized loops. Every earlier lowering pass sees an ordinary vectorized loop, so sliding window, storage folding, unrolling and bounds inference are untouched.
• vectorize_loops vectorizes a SWAR loop as usual, then lowers it as SWAR (new VectorizeSWAR.{h,cpp}). The lanes of each vector are packed into one GPR (up to the target's word, 32 or 64 bits), and lane-wise ops become integer instructions:
  • dense loads and stores as whole words;
  • bitwise ops;
  • uniform and per-lane-constant shifts;
  • masked-carry adds and subtracts;
  • power-of-two multiplies, divides and mods;
  • blends for selects.
  • Predicated loads and stores in vector tails, and conditions that vary across the lanes at runtime, are done one lane at a time.
• Consumers vectorized in SIMD registers read the result as usual. The hand-off is a register move, e.g. fmov s0, w8; mov.s v0[1], w9 on AArch64.
• A SWAR loop nested inside a SIMD loop (in the same Func, or in a Func computed at the SIMD loop) is lowered first, to scalar code on words, which the SIMD loop then vectorizes like any other scalars. This leaves dense vector accesses in the SIMD loop's body. To support them, VectorSubs::widen now repeats each lane of a vector in place (matching how ramps, broadcasts, loads and reinterprets are widened), and accesses to allocations inside the SIMD loop keep ramp indices dense. The generated code for the four-block decode above is uint32x4 arithmetic on four 4-byte row loads.
• vectorize_loops now takes the Target, which gives it the width of the general-purpose registers.
• MemoryType::GPR: like Register, it needs a constant size and is never moved to the heap. In LLVM codegen, each dense vector load and store of it is a single integer load or store, bitcast to and from the vector. The C backend treats it like Register.

User errors:

• an operation that can't be done lane-wise in a GPR (naming the Func, loop and expression, and suggesting VectorizeIn::SIMD);
• a vector too wide for a GPR;
• a SIMD-vectorized loop nested inside a SWAR one;
• a SWAR loop on a GPU or other device API;
• a dense vector access to a MemoryType::GPR allocation wider than a GPR;
• MemoryType::GPR with a dynamic size, or inside a GPU loop.

Benchmarks. These were measured on an Apple M3 Max with one thread, using apps/ggml (#9511, higher in this stack).

q4_K's 6-bit scale decode (SWAR compute):

k.in(readers).compute_at(acc, ryo).split(k, kw, kb, 4).unroll(kw).vectorize(kb, 4, VectorizeIn::SWAR);

• Decode: it compiles to pure w-register ldp/and/orr/stp code with no SIMD instructions.
• q4_K vec_dot: 130 → 118 instructions per block (76 → 72 SIMD), 7–12% faster (1.04× GGML's Neon kernel).
• q5_K: vec_dot is 4–7% faster, and its single-row mul_mat 4–9%.

Breaking changes

None. This adds two enum values (ForType::VectorizedSWAR and MemoryType::GPR), a new enum (VectorizeIn) and vectorize overloads, and makes vectorize_loops (internal) take a Target.

Checklist

• Tests added or updated: correctness_vectorize_swar, correctness_vectorize_swar_errors
• Documentation updated (VectorizeIn, MemoryType::GPR and Func::vectorize docs)
• Python bindings updated (hl.VectorizeIn, hl.MemoryType.GPR, vectorize overloads)
• Benchmarks are included here if the change is intended to affect performance.
• Commits include AI attribution where applicable (see Code of Conduct)

[mcourteaux]

Conclusion from dev-meeting: SWAR should be a loop-type, to be able to vectorize a SWAR loop. If it's triggered by a memory type, you say "put it in GPR", which is fundamentally not compatible with then vectorizing it into real vector registers.

[codecov]

Codecov Report

❌ Patch coverage is 65.16129% with 216 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.96%. Comparing base (c7324f6) to head (824c5e8).

Files with missing lines	Patch %	Lines
src/VectorizeGPR.cpp	59.96%	139 Missing and 62 partials ⚠️
src/VectorizeLoops.cpp	91.80%	2 Missing and 3 partials ⚠️
src/BoundSmallAllocations.cpp	90.62%	2 Missing and 1 partial ⚠️
src/IRPrinter.cpp	0.00%	2 Missing and 1 partial ⚠️
src/ScheduleFunctions.cpp	70.00%	2 Missing and 1 partial ⚠️
src/CodeGen_C.cpp	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@                        Coverage Diff                         @@
##           alexreinking/const-lut-shuffle    #9596      +/-   ##
==================================================================
+ Coverage                           72.76%   72.96%   +0.20%     
==================================================================
  Files                                 263      264       +1     
  Lines                               82458    83065     +607     
  Branches                            20164    20365     +201     
==================================================================
+ Hits                                60001    60609     +608     
- Misses                              16006    16076      +70     
+ Partials                             6451     6380      -71     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[abadams]

Is there a test case that reaches this? If a vectorized Func stored in a GPR is an error earlier in lowering, I'm not sure how we could reach here. Unaligned struct loads lowered as small vector loads?

[abadams]

Suggested renamings:

GPR -> Scalar (we can promise it's lowered as a scalar, but we can't promise where LLVM will place it)
VectorizeIn:: -> Vector::