cuda.core: Add conditional retry when default mempool reservation fails with OOM

[juenglin]

Addresses #2381

Device.memory_resource could fail with CUDA_ERROR_OUT_OF_MEMORY on a device with
ample free memory. A memory pool's virtual address reservation is only returned when
the pool is torn down, and cuMemPoolDestroy waits on the stream-ordered
cuMemFreeAsync calls for the pool's outstanding allocations. While those frees sit
queued behind unfinished work, the address space is recoverable but not yet
recovered — enough to fail the next pool lookup wherever address space is scarce.
get_device_mempool now drains the context once and retries:

• Retries only on CUDA_ERROR_OUT_OF_MEMORY; every other status returns unchanged.
• Retries at most once, with a cuCtxSynchronize
• The happy path is untouched; the synchronize only runs where we were about to raise.
• If the synchronize itself fails, the original OOM is preserved. The handle layer has
  a single thread-local error slot, so reporting the drain's failure instead would
  misattribute why the pool was unavailable.

Scope and cost

• Best effort across devices. The drain covers the current context, but
  get_device_mempool takes a device_id. When that device is not current, the retry
  retires unrelated work and will not recover the target pool. It then falls back to
  reporting the original OOM, so this is a missed recovery rather than a wrong result.
• Bounded on the success path. The driver creates the default pool lazily, and once
  it exists the lookup cannot fail. A retry that succeeds therefore costs one
  synchronize per device per process, not one per allocation. A retry that fails can
  recur, so a run that stays exhausted pays one drain per attempt — cheap on an idle
  GPU, but not free.
• Not a blanket fix. Only the default-pool path retries. cuMemPoolCreate (used by
  pinned, managed, and options-based resources) and cuGraphAddMemAllocNode can fail
  the same way and are unchanged.

Testing

cuda_core/tests/test_mempool_oom_retry.py reproduces the failure and verifies recovery
in a single run: it exhausts the address space, blocks the deallocation stream on an
event behind a slow kernel, drops the pools, asserts the un-retried lookup still fails,
then asserts the retried one succeeds and actually blocked. It runs in a spawned child
process because the default pool is created lazily — once it exists the lookup cannot
fail, so a process that already touched it could not reproduce this. Confirmed to fail
against main for the right reason. It skips on machines whose address space is too
large to exhaust.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[juenglin]

/ok to test

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2474/
https://nvidia.github.io/cuda-python/pr-preview/pr-2474/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2474/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2474/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[juenglin]

/ok to test

[juenglin]

/ok to test

[juenglin]

/ok to test

[juenglin]

/ok to test