Repository navigation
Compile less of cuTile's own code per kernel configuration - #345
Merged
Merged
Conversation
The value-numbering table was a `Dict{Tuple, SSAValue}` keyed by
`(func, type, operands...)`, whose concrete type differs per intrinsic and
operand types, so every new combination compiled `get`, `setindex!` and
`hashindex` again (13-23 ms each). Key it by a `Vector{Any}` instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The closure pushed onto `sv.tasks` captured the tfunc's return type, a
`Type{Tile{T, S}}`, so its type, and its compiled code, differed for every
element type and tile shape (about 8 ms each).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`AliasGroups`' concrete type differs with every grouping, and the keyword functions it passes through (`tile_job`, `emit_kernel!`, `run_passes!`, `analyze_aliases`) were inferred and compiled again for each one: about 70 ms of `emit_kernel!` per grouping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An `ArraySpec` is a singleton whose type differs with every alignment, divisibility and layout, so `arg_chain`, the tensor view shape and stride helpers, `ArraySpec`'s `getproperty`, and the hint closures in `make_tensor_view`'s codegen compiled again for each new spec: about 35 ms for a kernel that only changes its element type.
`emit_tile`'s bytecode-writing closure captured `num_ctas`, `occupancy` and `num_worker_warps`, each `nothing` or an `Int`, so its type and the `write_bytecode!` it is passed to compiled again for every combination (about 180 ms for a first `num_ctas=2`). Capture an `EntryHints` instead, which `emit_kernel!` built from them anyway, and don't specialize `emit_kernel!` on whether constant arguments are given.
`cufunction_compile`, `launch_job` and `tile_job` were specialized on the options' NamedTuple and the constant arguments' tuple, whose types differ with which options are set and with the alias grouping, and the overlap labels were computed in a `TileKernel` constructor and a comprehension specialized on the grouping. Together that compiled 50-250 ms again for each new `num_ctas`, `opt_level` or overlap pattern; it is now about 30 ms. With one signature left, `cufunction_compile` can also be precompiled, which the specialized version never was; that takes about 130 ms off the first kernel.
maleadt
marked this pull request as ready for review
November 5, 2026 07:56
Member
|
I pushed three additional commits. They fix more of the same kind of configuration-dependent compilation, mostly on the launch path, which your benchmark doesn't reach because it calls tile_job directly:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some of cuTile's own Julia code compiles again for every new kernel configuration (tile shape, element type, intrinsic and operand types, alias grouping, compile options), because the type of a value it handles depends on that configuration. This PR fixes six places where that happens:
CSE's value-numbering table (
transform/cse.jl) was aDict{Tuple, SSAValue}keyed by(func, type, operands...). The key's concrete type differs for each intrinsic and its operand types, so every new combination compiledget,setindex!,hashindexandisequalagain, at 13–23 ms each. The table is now keyed by aVector{Any}.The task that finishes a known call's inference (
abstract_call_known, Julia 1.12+) was a closure that captured the tfunc's return type, aType{Tile{T, S}}. The closure's type therefore differed for every element type and tile shape, and each one compiled again (about 8 ms). It is now a struct with untyped fields.Alias groups: the concrete type of
AliasGroupsdiffers for every grouping, andtile_job,emit_kernel!,run_passes!,analyze_aliasesandAliasAnalysiswere inferred again for each one (about 70 ms ofemit_kernel!). They are now marked@nospecializewithBase.@nospecializeinfer.Array specs (
intrinsics/views.jl,analysis/assume.jl,language/types.jl): anArraySpecis a singleton whose type differs with every alignment, divisibility and layout, andarg_chain, the tensor view shape and stride helpers,ArraySpec'sgetproperty, and closures over the spec inmake_tensor_view's codegen were compiled again for each one (about 35 ms when only the element type changes).Entry hints (
driver.jl,codegen/kernel.jl):emit_tile's bytecode-writing closure capturednum_ctas,occupancyandnum_worker_warps, eachnothingor anInt, so it andwrite_bytecode!compiled again for every combination (about 180 ms). It now captures oneEntryHints.The launch path (
launch.jl):cufunction_compile,launch_jobandtile_jobwere specialized on the options' NamedTuple and the constant arguments, and the overlap labels were computed in code specialized on the alias grouping, so every new option set or overlap pattern compiled 50–250 ms again.cufunction_compilenow has a single signature, so it is also precompiled, which takes about 130 ms off the first kernel.None of these changes what cuTile emits. The script's kernels, and 31 kernels from an MLP's and an attention layer's training steps, disassemble to identical Tile IR before and after. Only the debug info differs, because it records source paths.
Measurements
Julia 1.13.2 on a GB10 (sm_121). The script below compiles a small 2-D kernel through
compile_or_lookup, one new configuration at a time, in a fresh session where only cuTile is loaded. A new session runs cuTile's front end for every configuration even when the CUBIN is already in the disk cache, because that cache sits after the front end. So the Julia compile time is the same cold and warm; a warm session skips onlytileiras, about 110 ms per kernel.Float3264×64Float16Float64aandcaliasinga,bandcaliasingEach cell gives before → after. The Julia compile column spans the cold and warm runs, which agree. Kernels with more intrinsics and tile shapes save more.
Through
cuTile.cufunctionMeasured on an RTX 5080 (sm_120), Julia 1.13.2, with the CUBINs in the disk cache. 1–3 are the first three commits. The script is the one below with
ct.cufunction(kernel, tt; kwargs...)in place oftile_jobandcompile_or_lookup. Julia compile time:Float16opt_level=2num_ctas=2aandcaliasinga,bandcaliasingWhat remains for a new tile shape is mostly
cufunctionitself, which is specialized on the argument types so that launches infer, as in CUDA.jl.Script
Run it twice with the same
JULIA_OBJCACHE_PATH: the first run starts from an empty disk cache, and the second finds every CUBIN in it.Testing
On an RTX 5080 (Julia 1.13.2):
Pkg.test()passes, with 3088 pass and 1 broken.Pkg.test(): 2608 pass and 1 broken.device/randomerrors on this machine onmaintoo: LLVM can't select an SVE comparison for aBFloat16broadcast on the host (aarch64).🤖 Generated with Claude Code