CI: Improving Cache Hit Rates and Seeding

[paultiq]

Summary

After #559, there's a few more ways to improve caching and get the seeds to 100% hit rates:

• Use a stable --env-dir so that the Python compilation units are cacheable
• Use a stable build-dir so that the fanout builds can fully use the seed builds
• Raise the MACSX_DEPLOYMENT_TARGET so that the cp311 builds can use the seed builds

Current problems

1. cibuildwheel uses a random path dir which causes the Python compilation units to consistently miss. There hasn't been a great solution until yesterday (2026-08-27) when pypa/build 1.6.0 shipped: #1083 control the isolated build environment location w --env-dir. This feature will solve one specific issue: the 51 cache misses in the seed builds due to the random build path created by cibuildwheel.

2. Using a dynamic build-dir means that each build is contaminated by the cache_tag: the cache_tag ends up part of some ccache compile calls, which then aren't reused across builds.
   

   duckdb-python/pyproject.toml

   Line 84 in 6239d16

   build-dir = "build/{cache_tag}"

3. The seed cache is dependent on the MACOSX_DEPLOYMENT_TARGET, set here:
   

   duckdb-python/pyproject.toml

   Line 510 in 6239d16

   environment = { MACOSX_DEPLOYMENT_TARGET = "10.14" }

   
   but cibuildwheel will autoadjust that to 10.15 for some builds and ignore your input: cibuildwheel MACOSX_DEPLOYMENT_TARGET. This causes a mismatch between cp311 and other builds, causing cp311 to miss.

Alternative: If you need to support cp311 on 10.14, then could also move it up to seed so it generates its own cache.

Changes

• Use --env-dir and pypa/build 1.6.0 to eliminate the random Python dir.
• Set build-dir to a constant when running cibuildwheel
• Use a deployment target that works for all builds: 10.15.
• add verbose logging to help with future debugging: you need the full path to see what's changed
  Versions
• cibuildwheel to 4.2.0 (latest)
• nanobind < 3.0 (mentioned elsewhere, needed for builds due to a breaking change)

Current State Notes

• seeds get about 88% hits. The 51 misses are the Python misses fixed by the --env-dir change above
• The fanouts are around ~50% due to the {cache_tag}.
• The cp311 macos x86_64 builds have a 0% cache hit rate due to the MACOSX_DEPLOYMENT_TARGET

Impact

• 100% cache hits on seeds vs 88%
• ~88% cache hits on the fanout vs ~51%
• Reference: https://github.com/duckdb/duckdb-python/actions/runs/32117618656/job/95671628960
• The forked seeds are more or less the same time: the cache hits don't have a significant runtime difference, since it's dominated by test time and the 51 python units are fairly fast.
• The forked fanout/matrix builds are 5-10 minutes faster... the caching makes a big difference.

Futures / other optimizations worth considering

• Two tests are about 30% of total runtime: test_executemany_leak and test_materialized_relation[10000000]
• Don't install ccache twice on Windows and macosx. This is 30sec-1min.
• Don't install ccache from EPEL on Linux, use curl & cache it. I saw this take up to 2 minutes.
• Set -n auto or -n 2 for pytests.. there's a few places where tmppath fixtures will be needed, but this cuts CI time down significantly.
• Bump setup-uv and uv-version to latest, and enable uv cache.

[evertlammerts]

Thanks!