[mcourteaux]

Copies from device to host on Vulkan were extremely slow on GPUs with host-visible VRAM: about 22 MB/s on an RTX 4070, against several GB/s on CUDA.

Cause

The staging buffer of copy_to_host and of device-to-host buffer_copy is requested with MemoryVisibility::DeviceToHost and MemoryCaching::UncachedCoherent. For DeviceToHost, preferred_flags_for_memory_properties adds VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT, and nothing asks for HOST_CACHED. On devices that expose host-visible VRAM (the PCIe BAR window, or all of VRAM with resizable BAR) that preference selects VRAM for the staging buffer. The GPU's copy into it is fast, but the CPU then reads it through an uncached mapping across the bus, one non-posted read after another.

Memory types on the RTX 4070 used for testing (resizable BAR off, 256 MiB BAR):

type	heap	flags	CPU read, measured	CPU write, measured
3	system RAM	HOST_VISIBLE | HOST_COHERENT | HOST_CACHED	11.7 GB/s	13.3 GB/s
4	246 MiB VRAM (BAR)	DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT	22.1 MB/s	5.8 GB/s

The old request picked type 4. When that heap was full, the fallback took the first host-visible type (uncached system RAM), which is also slow to read.

Fix

For memory the host reads, prefer HOST_CACHED (and coherent) memory:

• DeviceToHost now prefers HOST_CACHED instead of DEVICE_LOCAL;
• the two staging requests ask for MemoryCaching::CachedCoherent.

Devices without host-cached memory still fall back to any host-visible memory, as before. Uploads are unchanged.

Results (RTX 4070, Linux)

• A 48 MB copy_to_host: from ~2 s down to ~7.5 ms (6.4 GB/s).
• In an application exporting a 24 MP RAW photo through 13 Halide Vulkan pipelines, the device-to-host copies went from 530 ms to 2.3 ms per export. Halide's other threads also no longer wait behind those slow copies (the context lock is held during the copy), so the total went from 1011 ms to 402 ms.

Tests

Copy-related correctness tests on Vulkan (HL_JIT_TARGET=host-vulkan-vk_v13-vk_int8-vk_int16-vk_int64): async_copy_chain, async_device_copy, device_buffer_copies_with_profile, device_buffer_copy, device_copy_at_inner_loop, device_crop, device_slice, extern_stage_on_device, gpu_give_input_buffers_device_allocations, gpu_jit_explicit_copy_to_device, gpu_mixed_dimensionality, gpu_non_contiguous_copy, host_inside_gpu_loop, leak_device_memory: all pass (gpu_async_copy skips, CUDA-only).

🤖 Generated with Claude Code

https://claude.ai/code/session_011FRemi1N2Cn82dNxq2aZkF

[derek-gerstmann]

Thanks for digging into this! The existing pool selection was designed to select the largest available memory pool to avoid out-of-memory issues on constrained devices. However, your commit that adjusts the memory property flags to prefer the fast path for readbacks with an appropriate fallback is a big win overall. Approved with a few changes.

[derek-gerstmann]

In src/runtime/vulkan_memory.h:877 — Fallback memory selection can pick non-coherent HOST_VISIBLE memory when HOST_CACHED | HOST_COHERENT is unavailable.

• Mechanism: VulkanMemoryAllocator::allocate_block (src/runtime/vulkan_memory.h:714-785) runs two passes:
  1. Pass 1 (preferred): requires required_flags | preferred_flags. For (DeviceToHost, CachedCoherent), this is HOST_VISIBLE_BIT | HOST_CACHED_BIT | HOST_COHERENT_BIT.
  2. Pass 2 (fallback): requires only required_flags, which required_flags_for_memory_properties sets to HOST_VISIBLE_BIT alone (src/runtime/vulkan_memory.h:851).
• Why it matters: The Vulkan specification (§11.2) guarantees at least one HOST_VISIBLE | HOST_COHERENT memory type on every device, but does not guarantee a HOST_VISIBLE | HOST_CACHED | HOST_COHERENT type. On devices where cached host memory is not coherent (e.g., mobile/ARM SoCs exposing separate HOST_VISIBLE | HOST_CACHED and HOST_VISIBLE | HOST_COHERENT types), Pass 1 finds no match and falls back to Pass 2 (HOST_VISIBLE only). If HOST_VISIBLE | HOST_CACHED (non-coherent) appears before HOST_VISIBLE | HOST_COHERENT, Pass 2 selects non-coherent memory even though Halide never calls vkInvalidateMappedMemoryRanges (or vkFlushMappedMemoryRanges), risking stale CPU cache reads.
• Suggested Fix: In required_flags_for_memory_properties, require VK_MEMORY_PROPERTY_HOST_COHERENT_BIT whenever a host-visible allocation requests CachedCoherent or UncachedCoherent:

  switch (properties.caching) {
  case MemoryCaching::CachedCoherent:
  case MemoryCaching::UncachedCoherent:
      if (required_flags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) {
          required_flags |= VK_MEMORY_PROPERTY_HOST_COHERENT_BIT;
      }
      break;
  case MemoryCaching::Cached:
  case MemoryCaching::Uncached:
  case MemoryCaching::DefaultCaching:
      break;
  case MemoryCaching::InvalidCaching:
  default:
      error(user_context) << "VulkanMemoryAllocator: Unable to convert type! Invalid memory caching request!\n\t"
                          << "caching=" << halide_memory_caching_name(properties.caching) << "\n";
      return invalid_memory_type;
  };

[mcourteaux]

Good catch: the runtime never calls vkFlushMappedMemoryRanges / vkInvalidateMappedMemoryRanges, so it relies on coherent memory. Done in a6d16a3 as you suggested: required_flags_for_memory_properties now requires HOST_COHERENT for host-visible memory when the caching is CachedCoherent or UncachedCoherent, so the fallback pass can no longer pick non-coherent memory. The spec guarantees a HOST_VISIBLE | HOST_COHERENT type, so the requirement can always be met.

[derek-gerstmann]

src/runtime/vulkan_memory.h:877 — preferred_flags_for_memory_properties sets VK_MEMORY_PROPERTY_HOST_CACHED_BIT unconditionally under case MemoryVisibility::DeviceToHost:, overriding explicit Uncached / UncachedCoherent requests.

• Mechanism: switch (properties.caching) at lines 887–900 already adds VK_MEMORY_PROPERTY_HOST_CACHED_BIT for MemoryCaching::CachedCoherent and MemoryCaching::Cached. Setting VK_MEMORY_PROPERTY_HOST_CACHED_BIT unconditionally in switch (properties.visibility) under case MemoryVisibility::DeviceToHost: is redundant when caching is CachedCoherent (which both vulkan.cpp call sites now pass), and causes (DeviceToHost, UncachedCoherent) or (DeviceToHost, Uncached) to still prefer HOST_CACHED_BIT.
• Suggested Fix: Either only default DeviceToHost to HOST_CACHED_BIT when properties.caching == MemoryCaching::DefaultCaching, or let DeviceToHost fall through like HostToDevice so properties.caching alone controls cache flags:

  case MemoryVisibility::DeviceToHost:
      if (properties.caching == MemoryCaching::DefaultCaching) {
          preferred_flags |= VK_MEMORY_PROPERTY_HOST_CACHED_BIT;
      }
      break;

[mcourteaux]

Agreed, done in a6d16a3: DeviceToHost now only prefers HOST_CACHED when the caching is DefaultCaching; the two staging requests pass CachedCoherent, so the caching switch decides by itself. Re-measured on the RTX 4070 with both changes: a 48 MB copy_to_host takes ~8 ms (~6.3 GB/s), against 6.8 s (7 MB/s) on main. The 70 Vulkan gpu/device/async/copy correctness tests pass (61 pass, 9 skip for other backends).