Skip to content

fix(gpu_test): isolate concurrent test sessions - #91

Open
tetsuo-cpp wants to merge 3 commits into
canonfrom
fix/concurrent-gpu-tests
Open

fix(gpu_test): isolate concurrent test sessions#91
tetsuo-cpp wants to merge 3 commits into
canonfrom
fix/concurrent-gpu-tests

Conversation

@tetsuo-cpp

@tetsuo-cpp tetsuo-cpp commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • assign each GPU pytest session a unique Vast.ai label and bind it to the instance ID returned by the current SDK
  • inventory existing WarpForth instances at startup without deleting resources owned by other runs
  • generate a per-session Ed25519 identity and attach only its public key to the owned Vast container, so fresh orbs and CI need only VASTAI_API_KEY
  • reconcile every ambiguous create response by exact label and verify teardown after scoped destroy requests
  • add mocked concurrency/lifecycle tests, run them in CI, and document parallel-run costs and orphan recovery
  • migrate from the legacy vastai-sdk package to the current official vastai package

Verification

  • env -u VASTAI_API_KEY uv run pytest -q -m "not gpu" — 8 passed, 37 deselected
  • uv run ruff check gpu_test/
  • uv run ruff format --check gpu_test/
  • cmake --build build --target check-warpforth — 114 passed
  • uv lock --check
  • git diff --check
  • paid Vast.ai smoke test from a fresh orb: two independent pytest processes ran test_addition and test_subtraction concurrently; both attached distinct per-instance SSH keys, reached SSH readiness, compiled the runner, executed their kernels, and passed
  • final Vast inventory confirmed every test-created instance was absent and the pre-existing account instance remained untouched

The paid smoke test exposed a Vast response that reported failure after creating a contract; the final follow-up commit handles all responses without an authoritative ID as ambiguous and reconciles by exact label rather than issuing another create request.

@tetsuo-cpp
tetsuo-cpp force-pushed the fix/concurrent-gpu-tests branch from f4bbb64 to 7c804c8 Compare August 12, 2026 04:08
Comment thread gpu_test/conftest.py Outdated
@tetsuo-cpp
tetsuo-cpp force-pushed the fix/concurrent-gpu-tests branch from 7c804c8 to effbc1d Compare August 12, 2026 04:10
@tetsuo-cpp tetsuo-cpp changed the title fix(gpu): isolate concurrent test sessions fix(gpu_test): isolate concurrent test sessions Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant