Skip to content

Compile less of cuTile's own code per kernel configuration - #345

Merged
maleadt merged 6 commits into
mainfrom
ao/fewer-specializations
Nov 5, 2026
Merged

maleadt merged 6 commits into
mainfrom
ao/fewer-specializations

Conversation

@AntonOresten

@AntonOresten AntonOresten commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Some of cuTile's own Julia code compiles again for every new kernel configuration (tile shape, element type, intrinsic and operand types, alias grouping, compile options), because the type of a value it handles depends on that configuration. This PR fixes six places where that happens:

  1. CSE's value-numbering table (transform/cse.jl) was a Dict{Tuple, SSAValue} keyed by (func, type, operands...). The key's concrete type differs for each intrinsic and its operand types, so every new combination compiled get, setindex!, hashindex and isequal again, at 13–23 ms each. The table is now keyed by a Vector{Any}.

  2. The task that finishes a known call's inference (abstract_call_known, Julia 1.12+) was a closure that captured the tfunc's return type, a Type{Tile{T, S}}. The closure's type therefore differed for every element type and tile shape, and each one compiled again (about 8 ms). It is now a struct with untyped fields.

  3. Alias groups: the concrete type of AliasGroups differs for every grouping, and tile_job, emit_kernel!, run_passes!, analyze_aliases and AliasAnalysis were inferred again for each one (about 70 ms of emit_kernel!). They are now marked @nospecialize with Base.@nospecializeinfer.

  4. Array specs (intrinsics/views.jl, analysis/assume.jl, language/types.jl): an ArraySpec is a singleton whose type differs with every alignment, divisibility and layout, and arg_chain, the tensor view shape and stride helpers, ArraySpec's getproperty, and closures over the spec in make_tensor_view's codegen were compiled again for each one (about 35 ms when only the element type changes).

  5. Entry hints (driver.jl, codegen/kernel.jl): emit_tile's bytecode-writing closure captured num_ctas, occupancy and num_worker_warps, each nothing or an Int, so it and write_bytecode! compiled again for every combination (about 180 ms). It now captures one EntryHints.

  6. The launch path (launch.jl): cufunction_compile, launch_job and tile_job were specialized on the options' NamedTuple and the constant arguments, and the overlap labels were computed in code specialized on the alias grouping, so every new option set or overlap pattern compiled 50–250 ms again. cufunction_compile now has a single signature, so it is also precompiled, which takes about 130 ms off the first kernel.

None of these changes what cuTile emits. The script's kernels, and 31 kernels from an MLP's and an attention layer's training steps, disassemble to identical Tile IR before and after. Only the debug info differs, because it records source paths.

Measurements

Julia 1.13.2 on a GB10 (sm_121). The script below compiles a small 2-D kernel through compile_or_lookup, one new configuration at a time, in a fresh session where only cuTile is loaded. A new session runs cuTile's front end for every configuration even when the CUBIN is already in the disk cache, because that cache sits after the front end. So the Julia compile time is the same cold and warm; a warm session skips only tileiras, about 110 ms per kernel.

new configuration Julia compile wall, empty disk cache wall, CUBIN in disk cache
the first kernel, Float32 64×64 313–315 → 259–274 ms 489 → 435 ms 364 → 308 ms
each new tile shape (128×32, 32×128, 16×256) 14–16 → 0–2 ms 137–138 → 112–136 ms 22–25 → 8–11 ms
Float16 130–131 → 88–94 ms 247 → 191 ms 141 → 104 ms
Float64 197–198 → 145–159 ms 348 → 283 ms 208 → 170 ms
a and c aliasing 180–182 → 86–96 ms 315 → 187 ms 182 → 98 ms
a, b and c aliasing 115–116 → 21–24 ms 199 → 129 ms 117 → 25 ms

Each cell gives before → after. The Julia compile column spans the cold and warm runs, which agree. Kernels with more intrinsics and tile shapes save more.

Through cuTile.cufunction

Measured on an RTX 5080 (sm_120), Julia 1.13.2, with the CUBINs in the disk cache. 1–3 are the first three commits. The script is the one below with ct.cufunction(kernel, tt; kwargs...) in place of tile_job and compile_or_lookup. Julia compile time:

new configuration after 1–3 after 1–6
the first kernel 386–418 ms 270 ms
a new tile shape 46–52 ms 36 ms
Float16 75–76 ms 61 ms
opt_level=2 66–68 ms 28 ms
num_ctas=2 248–255 ms 29 ms
a and c aliasing 353–366 ms 102 ms
a, b and c aliasing 152–176 ms 22 ms

What remains for a new tile shape is mostly cufunction itself, which is specialized on the argument types so that launches infer, as in CUDA.jl.

Script

Run it twice with the same JULIA_OBJCACHE_PATH: the first run starts from an empty disk cache, and the second finds every CUBIN in it.

using Printf
import cuTile as ct
Base.cumulative_compile_timing(true)
ctime() = Base.cumulative_compile_time_ns()[1] / 1e9

function kernel(a::ct.TileArray{T, 2}, b::ct.TileArray{T, 2}, c::ct.TileArray{T, 2},
                tx::Int, ty::Int) where {T}
    i, j = ct.bid(1), ct.bid(2)
    x = ct.load(a; index = (i, j), shape = (tx, ty))
    y = ct.load(b; index = (i, j), shape = (tx, ty))
    ct.store(c; index = (i, j), tile = exp.(x) .* y .+ x)
    return
end

array(T) = ct.TileArray(Ptr{T}(256), Int32.((1024, 1024)), Int32.((1, 1024)))

function compile(T, tx, ty, groups)
    tt = Tuple{typeof(array(T)), typeof(array(T)), typeof(array(T)), ct.Constant{Int, tx}, ct.Constant{Int, ty}}
    c, t = ctime(), time()
    job = ct.invoke_frozen(ct.tile_job, kernel, tt; sm_arch = v"12.0", alias_groups = groups)
    ct.invoke_frozen(ct.compile_or_lookup, job)
    time() - t, ctime() - c
end

configs = [
    (Float32, 64, 64, ()),              # first: everything compiles once
    (Float32, 128, 32, ()),             # new tile shapes
    (Float32, 32, 128, ()),
    (Float32, 16, 256, ()),
    (Float16, 64, 64, ()),              # new element types
    (Float64, 64, 64, ()),
    (Float32, 64, 64, (((2, ()), (4, ())),)),            # `a` and `c` alias
    (Float32, 64, 64, (((2, ()), (3, ()), (4, ())),)),   # `a`, `b` and `c` alias
]
for (T, tx, ty, groups) in configs
    wall, julia = compile(T, tx, ty, groups)
    @printf("%-8s %3d×%-3d %-18s %6.3f s   julia %6.3f s\n", T, tx, ty,
            isempty(groups) ? "no aliasing" : "$(length(groups[1])) arrays alias", wall, julia)
end

Testing

On an RTX 5080 (Julia 1.13.2): Pkg.test() passes, with 3088 pass and 1 broken.

Pkg.test(): 2608 pass and 1 broken. device/random errors on this machine on main too: LLVM can't select an SVE comparison for a BFloat16 broadcast on the host (aarch64).

🤖 Generated with Claude Code

AntonOresten and others added 6 commits October 29, 2026 23:05
The value-numbering table was a `Dict{Tuple, SSAValue}` keyed by
`(func, type, operands...)`, whose concrete type differs per intrinsic and
operand types, so every new combination compiled `get`, `setindex!` and
`hashindex` again (13-23 ms each). Key it by a `Vector{Any}` instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The closure pushed onto `sv.tasks` captured the tfunc's return type, a
`Type{Tile{T, S}}`, so its type, and its compiled code, differed for every
element type and tile shape (about 8 ms each).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`AliasGroups`' concrete type differs with every grouping, and the keyword
functions it passes through (`tile_job`, `emit_kernel!`, `run_passes!`,
`analyze_aliases`) were inferred and compiled again for each one: about
70 ms of `emit_kernel!` per grouping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An `ArraySpec` is a singleton whose type differs with every alignment,
divisibility and layout, so `arg_chain`, the tensor view shape and stride
helpers, `ArraySpec`'s `getproperty`, and the hint closures in
`make_tensor_view`'s codegen compiled again for each new spec: about 35 ms
for a kernel that only changes its element type.
`emit_tile`'s bytecode-writing closure captured `num_ctas`, `occupancy`
and `num_worker_warps`, each `nothing` or an `Int`, so its type and the
`write_bytecode!` it is passed to compiled again for every combination
(about 180 ms for a first `num_ctas=2`). Capture an `EntryHints`
instead, which `emit_kernel!` built from them anyway, and don't specialize
`emit_kernel!` on whether constant arguments are given.
`cufunction_compile`, `launch_job` and `tile_job` were specialized on the
options' NamedTuple and the constant arguments' tuple, whose types differ
with which options are set and with the alias grouping, and the overlap
labels were computed in a `TileKernel` constructor and a comprehension
specialized on the grouping. Together that compiled 50-250 ms again for
each new `num_ctas`, `opt_level` or overlap pattern; it is now about 30 ms.
With one signature left, `cufunction_compile` can also be precompiled,
which the specialized version never was; that takes about 130 ms off the
first kernel.
@maleadt
maleadt marked this pull request as ready for review November 5, 2026 07:56
@maleadt maleadt changed the title Compile less of the code generator per kernel configuration Compile less of cuTile's own code per kernel configuration Nov 5, 2026
@maleadt

maleadt commented Nov 5, 2026

Copy link
Copy Markdown
Member

I pushed three additional commits. They fix more of the same kind of configuration-dependent compilation, mostly on the launch path, which your benchmark doesn't reach because it calls tile_job directly:

  • The code generator also specialized on the ArraySpec, so each new element type or alignment recompiled part of it (~35 ms, visible in your benchmark too).
  • emit_tile's closure captured the entry hints, each nothing or an Int, so the first num_ctas=2 recompiled the bytecode writer (~180 ms). It now captures one EntryHints.
  • cufunction_compile, launch_job and tile_job specialized on the options and the constant arguments, and the overlap labels on the grouping. Through ct.cufunction, a new option set or overlap pattern now costs ~30 ms instead of 50–250 ms. With one signature left, cufunction_compile is also precompiled, which takes ~130 ms off the first kernel.

@maleadt
maleadt merged commit 9fd3a34 into main Nov 5, 2026
5 checks passed
@maleadt
maleadt deleted the ao/fewer-specializations branch November 5, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants