Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,6 @@ jobs:

- name: Ruff check
run: uv run ruff check gpu_test/ && uv run ruff format --check gpu_test/

- name: Test GPU infrastructure
run: uv run pytest -q -m "not gpu"
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,3 +128,20 @@ cmake --build build --target check-warpforth
# Run end-to-end GPU tests (requires Vast.ai API key)
VASTAI_API_KEY=xxx uv run pytest -v -m gpu
```

Each GPU test process rents its own uniquely labeled Vast.ai instance, so GPU
tests can run concurrently from separate checkouts or Amp threads using the
same Vast.ai account. Each process logs existing instances whose labels start
with `warpforth-test-` but never destroys instances owned by another run.
Running multiple processes therefore increases Vast.ai usage and cost;
pytest-xdist also rents one instance per worker.

Each process generates an ephemeral SSH key and attaches only its public key to
the instance it owns. The private key remains in a local temporary directory
until teardown, so ad hoc machines and CI require only `VASTAI_API_KEY`; no
account-level Vast.ai SSH key configuration is required.

Normal teardown verifies that the process's instance was removed. A hard kill
or machine failure can still bypass cleanup. If startup logs report a possible
orphan, inspect the instance ID with `uv run vastai show instances` and remove
only the confirmed orphan with `uv run vastai destroy instance <id>`.
Loading
Loading