[mcourteaux]

Stacked on #9676 (base branch mcourteaux/vulkan-dispatch-overhead). It addresses the first two items of that PR's "What's left".

Why

A standalone raw-Vulkan microbenchmark on the RTX 4070 shows where the time goes after #9676:

	µs
kernel execution (GPU timestamps)	~2
one dispatch per command buffer, back to back	14–18
16 command buffers in one vkQueueSubmit	11–13.5
16–64 dispatches in one command buffer (compute barrier between each)	1.3–2.2
same, with an ALL_COMMANDS barrier between each	7.6–9
submit + vkWaitForFences	46–62
submit + spin on vkGetFenceStatus	16–20

So:

• Every command buffer costs ~11–16 µs on the GPU side, however it is submitted. The only way below that is to put several operations into one command buffer.
• Most of the "≈35–40 µs driver latency" in [vulkan] Reduce the fixed per-operation cost of the runtime #9676 is the wake-up from a blocking fence wait.

Changes

1. Record several operations into one submission. Kernel launches, uploads, clears and device-to-device copies are recorded into the submission currently being recorded (at most 32 commands). It is submitted:

   • when it is full,
   • when the GPU is running out of work: at most one earlier submission is still running (this also applies when a pipeline returns, in halide_vulkan_finalize_kernels),
   • before anything waits for submitted work (copies to the host, device_sync, frees that wait, pipeline recreation, device_release).

   The descriptor pool, scalar-argument buffers and staging-buffer list of a submission now hold one entry per command. The descriptor pool grows when a launch needs more storage buffers than it has room for.

2. Narrower barrier between commands. Each command is preceded by a compute|transfer → compute|transfer barrier, instead of the ALL_COMMANDS barrier at the start of each command buffer. The runtime records only compute and transfer work, and host writes before vkQueueSubmit are visible to the work it submits. Inside a batch, the ALL_COMMANDS barrier costs ~5 µs per command (table above).

3. Spin before blocking on fences. Waits check the fences with vkGetFenceStatus for up to 100 µs before falling back to vkWaitForFences. This applies both to waits for results and to waits for a ring slot.

Results (performance_gpu_dispatch_overhead, RTX 4070, µs unless noted)

	#9676	this PR	CUDA (#9676)
launch, back to back	15.3	4.1–5.6	3.0
launch + device_sync	47–57	16.5–19	6.4
idle device_sync	0.07–0.11	0.04–0.08	0.36
upload 64 KiB	18.1	8.1–9.0 (one outlier at 13)	12
upload 64 KiB + launch	33.0	12.8–13.7	13
malloc + free 64 KiB	12.8	1.8–2.5	0.6
upload 48 MB	7.7 ms	7.5–8.0 ms	4.7 ms

Both columns were measured back to back on the same build; only the runtime sources differ.

Behaviour to be aware of

• After a pipeline returns, its last commands may still be recorded but not yet submitted, if the GPU still had two or more submissions to run. They are submitted by the next Vulkan runtime call, a wait (halide_device_sync, copy to host), or once the GPU is running out of work. Results were already only guaranteed after halide_device_sync ([vulkan] Reduce the fixed per-operation cost of the runtime #9676). If the application does long CPU work right after a pipeline without calling into the runtime, that tail runs later than it would with [vulkan] Reduce the fixed per-operation cost of the runtime #9676 alone. Flushing unconditionally in halide_vulkan_finalize_kernels would avoid this, but it gives up batching across pipeline calls: with one launch per call, the back-to-back case then stays at ~15 µs.
• Waits burn up to 100 µs of CPU time before blocking.

Tests

• New correctness_gpu_many_launches_without_sync: long chains of dependent launches with no syncs (filling batches and wrapping the ring), uploads interleaved with launches, launches with 26 buffers each (exhausting descriptor pools mid-batch), and buffers freed right after the launches that use them. It passes on Vulkan and CUDA. With the barrier between commands disabled, it fails on Vulkan in 3 of 3 runs.
• On the RTX 4070 with HL_JIT_TARGET=host-vulkan-vk_v13-vk_int8-vk_int16-vk_int64, run directly because ctest forces HL_JIT_TARGET=host: a 237-test correctness subset gives 227 pass, 10 skip (inapplicable to the target) and 0 fail. The subset covers gpu, device, async, atomics, vulkan, shared memory, tuples, extern stages, gpu_object_lifetime_*, leak_device_memory, reductions, specialization and similar.

🤖 Generated with Claude Code

https://claude.ai/code/session_011FRemjkLh26DpFGTumbznR

[derek-gerstmann]

PR Review: [vulkan] Batch operations into one submission, and spin before blocking on fences

Summary

Batches up to 32 Vulkan operations (compute dispatches, buffer clears, host-to-device uploads, and device-to-device copies) into a single VulkanSubmission command buffer, narrows inter-command barriers from ALL_COMMANDS to COMPUTE_SHADER | TRANSFER, and spins on vkGetFenceStatus for up to 100 µs before blocking in vkWaitForFences. Scope matches the PR description across 1 commit and 5 files (+437/-134). Overall risk is Medium-High due to subtle interactions between deferred command buffer submission, mid-recording scalar buffer allocation, and RegionAllocator buffer destruction when splitting/coalescing released regions.

• Review mode: standard

Change Surface

• 21 symbols changed/added (vk_wait_for_fences, vk_release_staging_regions, vk_retire_submission, vk_create_submission_queue, vk_submit, vk_flush_submissions, vk_wait_for_submissions, vk_destroy_submission_queue, vk_start_submission, vk_begin_submission, vk_submission_is_done, vk_flush_submissions_if_gpu_needs_work, vk_end_submission, vk_attach_staging_region, vk_throttle_staging, vk_create_submission_descriptor_set, vk_submission_args_region, vk_create_descriptor_pool, halide_vulkan_finalize_kernels, halide_vulkan_copy_to_device, halide_vulkan_run)
• Starred (high-impact): halide_vulkan_run, halide_vulkan_copy_to_device, halide_vulkan_finalize_kernels, vk_begin_submission, vk_end_submission, vk_wait_for_submissions
• Surfaces touched: src/runtime/vulkan.cpp, src/runtime/vulkan_internal.h, src/runtime/vulkan_resources.h, test/correctness/CMakeLists.txt, test/correctness/gpu_many_launches_without_sync.cpp
• Scope: match

Blast Radius

• Checked all callers of vk_create_descriptor_pool, vk_begin_submission, vk_end_submission, vk_create_submission_descriptor_set, and vk_wait_for_submissions across src/runtime/vulkan*.
• Unupdated callers: VulkanMemoryAllocator::collect (src/runtime/vulkan_memory.h:510) calls vk_wait_for_submissions(user_context, this, false) assuming it only waits on already-submitted work; in this PR vk_wait_for_submissions flushes submission_queue->recording, which breaks if collect() is triggered inside halide_vulkan_run while a submission is mid-recording.
• Additionally, when halide_can_reuse_device_allocations is true, halide_vulkan_device_free (src/runtime/vulkan.cpp:162) marks regions Available while unsubmitted commands in submission_queue->recording (or in-flight submissions) still reference their VkBuffer. If a subsequent allocator->reserve() splits (RegionAllocator::split_block_region) or coalesces (BlockAllocator::reserve_memory_region -> RegionAllocator::collect -> coalesce_block_regions) that region, deallocate_block_region_if_unused immediately calls vkDestroyBuffer (vulkan_memory.h:1252) without flushing or waiting for submissions.
• Dangling references: none
• New orphans (dead_code): none

Prior Knowledge Surfaced

• none relevant

Test Gaps

• Varying-size device_free() + reallocation while commands referencing the freed buffer are still buffered in submission_queue->recording or in flight on the GPU.

Affected-Test Run

• Author verified correctness_gpu_many_launches_without_sync and 237-test GPU correctness subset on RTX 4070 (227 pass, 10 skip, 0 fail).

Lint

• git diff --check passed cleanly.

Security

• No issues found

Verdict

Request changes

Blocking items to resolve before merge:

1. Prevent VulkanMemoryAllocator::collect() from flushing and ending submission_queue->recording mid-command when vk_submission_args_region allocates a scalar uniform buffer during halide_vulkan_run.
2. Guard allocator == nullptr in vk_wait_for_submissions / vk_flush_submissions (and sibling submission queue helpers) so halide_vulkan_device_release does not dereference a null allocator when called before context creation.
3. Ensure splitting or coalescing an Available region in RegionAllocator does not call vkDestroyBuffer while unsubmitted (submission_queue->recording) or in-flight submissions still reference its VkBuffer.

[derek-gerstmann]

[HIGH] vk_wait_for_submissions now calls vk_flush_submissions, which ends (vkEndCommandBuffer) and submits (vkQueueSubmit) submission_queue->recording. However, halide_vulkan_run (src/runtime/vulkan.cpp:1174-1196) calls vk_submission_args_region -> vk_create_scalar_uniform_buffer -> allocator->reserve() after vk_begin_submission has started recording into submission. If vkCreateBuffer inside VulkanMemoryAllocator::allocate_region (src/runtime/vulkan_memory.h:1165) fails and triggers instance->collect(user_context) -> vk_wait_for_submissions(user_context, this, false), submission will be ended and submitted mid-recording before vk_record_dispatch_call (vulkan.cpp:1226) and vk_end_submission (vulkan.cpp:1235) run on it.

[mcourteaux]

Agreed. This branch is rebased onto #9676, whose deallocate_region now also waits for submissions, so this got wider than the vkCreateBuffer-failure path: a split while reserving the scalar-argument buffer would flush too. Fixed in 35c57dc: while vk_submission_args_region reserves or destroys an args buffer, the submission queue is marked as holding a half-recorded command. During that window vk_flush_submissions doesn't submit the recording submission (waits still wait for submitted work), and VulkanMemoryAllocator::collect skips collection. That window only splits, coalesces or frees uniform-buffer regions, which no recorded or submitted work uses once they are released.

🤖 Generated with Claude Code

[derek-gerstmann]

[HIGH] vk_flush_submissions (as well as vk_wait_for_submissions at line 409, vk_destroy_submission_queue at line 453, vk_flush_submissions_if_gpu_needs_work at line 612, and vk_throttle_staging at line 650) dereferences allocator->submission_queue without checking allocator == nullptr. When halide_vulkan_device_release (src/runtime/vulkan.cpp:302) is called before a Vulkan context has been initialized, halide_default_vulkan_acquire_context(..., create=false) returns halide_error_code_success with *allocator = nullptr, causing a null pointer dereference in vk_wait_for_submissions.

[mcourteaux]

Agreed. vk_wait_for_submissions and vk_destroy_submission_queue got the guard in #9676 (98bde6f, with correctness_gpu_device_release_idle), and 66de740 adds it to vk_flush_submissions, vk_flush_submissions_if_gpu_needs_work and vk_throttle_staging.

🤖 Generated with Claude Code

[derek-gerstmann]

[MEDIUM] Two issues with descriptor pool replacement here:

1. If submission->descriptor_pool is destroyed on line 509 and vk_create_descriptor_pool fails on line 515, submission->descriptor_pool is VK_NULL_HANDLE while submission->descriptor_pool_storage_buffer_count retains its old non-zero value (and doubles again on retry).
2. Because vk_start_submission only receives the single launch's storage_buffer_count, a batch flushed early at line 572 due to cumulative pool exhaustion (e.g. 9 launches of 26 buffers vs 256 capacity) passes storage_buffer_count = 26 <= 256 here, so the pool never grows and every such batch stays capped at 9 launches instead of 32.

[mcourteaux]

Agreed on both, fixed in 66de740. (1) The count is cleared before the pool is created, so a failed create leaves a null pool with a zero count; the next start creates one again. (2) When a batch is submitted because it ran out of descriptors, the next submission asks for at least twice that pool's storage-buffer count, so pools grow geometrically until full batches fit.

🤖 Generated with Claude Code

[derek-gerstmann]

[MEDIUM] If vk_begin_submission at line 538 returns nullptr (which in this PR can also happen when flushing a full batch via vk_submit fails), staging_region reserved at line 482 is not released via vk_release_region before returning.

[mcourteaux]

Agreed. Since the rebase onto #9676 (98bde6f), the copy functions hold their staging buffer in a ScopedStagingRegion that releases it on every early return, including a failed vk_begin_submission, unless it was handed to a submission.

🤖 Generated with Claude Code

[derek-gerstmann]

[HIGH] In this "freed buffers" test, tmp(W, H) has the exact same size on every iteration, so RegionAllocator always reuses the same BlockRegion and VkBuffer handle in-place. If subsequent allocations use a different size (e.g., alternating H and H / 2 or H * 2), RegionAllocator::reserve splits the released region (split_block_region at src/runtime/internal/region_allocator.h:453) or coalesces it (BlockAllocator::reserve_memory_region -> RegionAllocator::collect -> coalesce_block_regions at region_allocator.h:399), which calls deallocate_block_region_if_unused -> vkDestroyBuffer (vulkan_memory.h:1252) immediately on the host without flushing or waiting for submission_queue->recording or in-flight submissions that still reference that VkBuffer (violating VUID-vkDestroyBuffer-buffer-00922). Please also vary tmp's dimensions across iterations here to cover region split/coalesce.

[mcourteaux]

Agreed, done in 66de740: the freed buffer's height now cycles through H, 1.5H and 2H, and the launches use an H-row crop of it, so freed regions get split and coalesced while launches that use them are still recorded or in flight. The rebase also brings in correctness_gpu_free_while_in_flight from #9676, which covers the same case with larger size swings.

🤖 Generated with Claude Code