Skip to content

ci: run optional GPU tests via Vast.ai - #92

Open
tetsuo-cpp wants to merge 1 commit into
canonfrom
ci/vast-gpu-tests
Open

ci: run optional GPU tests via Vast.ai#92
tetsuo-cpp wants to merge 1 commit into
canonfrom
ci/vast-gpu-tests

Conversation

@tetsuo-cpp

@tetsuo-cpp tetsuo-cpp commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a manually triggered GPU test job to the existing CI workflow
  • use the GPU CI environment's VASTAI_API_KEY secret
  • keep GPU testing optional on PRs: normal PR runs skip the paid job
  • let repository users with write access select a PR branch via workflow_dispatch
  • wait for normal build/test/lint checks before renting a GPU
  • serialize paid GPU jobs and cap each job at 45 minutes
  • fail clearly if the environment secret is missing instead of silently skipping

Validation

  • actionlint .github/workflows/ci.yml
  • uv run pytest -m gpu --collect-only -q (37 tests collected)
  • uv run ruff check gpu_test/
  • uv run ruff format --check gpu_test/
  • cmake --build build --target warpforthc -j2

Copy link
Copy Markdown
Owner Author

Remaining setup for Vast.ai GPU CI

  1. Confirm this is an environment secret, not an environment variable. In Settings → Environments → GPU CI, VASTAI_API_KEY must appear under Environment secrets. The workflow reads secrets.VASTAI_API_KEY; a value under Environment variables (vars) will not work. The name and environment name are case-sensitive.
  2. Fund and verify the Vast.ai account. The key's account needs enough credit and permission to search offers, create instances, list instances, and destroy instances. The harness chooses one CUDA-capable GPU at no more than $0.50/hour and destroys it when the pytest session exits.
  3. Add an environment required reviewer before using this broadly. The GPU CI environment currently has no protection rules. A required reviewer gives a human a chance to inspect a PR before paid infrastructure starts and before its code receives the API key. Enable “Prevent self-review” if that fits the contributor model. Approval should only happen after reviewing all executable PR changes (workflow, Python tests, build files, and dependencies), because same-repository PR code can read the secret once approved.
  4. Optionally make the check required. In the canon branch protection/ruleset, add CI / gpu-test as a required status check if merges must wait for GPU execution. Fork PRs intentionally skip this job because GitHub must not disclose the Vast key to untrusted fork code.
  5. Use manual runs when needed. Actions → CI → Run workflow runs normal CI first and then the GPU job. GPU tests run automatically on same-repository PRs, but not again on the post-merge push to canon, avoiding duplicate Vast charges.
  6. Watch the first run and check for leftovers. Confirm all 37 tests execute (rather than skip) and verify no warpforth-test instance remains in Vast.ai afterward. The harness cleans up on normal failures and at process exit, but a hard GitHub runner termination can still warrant checking the Vast console.

The workflow serializes GPU jobs because the harness uses a shared warpforth-test instance label; this also prevents overlapping paid rentals.

Copy link
Copy Markdown
Owner Author

Update: GPU tests are now optional and manual-only

The PR was updated so ordinary pull_request runs always skip gpu-test. To test a PR after this workflow is merged into canon:

  1. Open Actions → CI → Run workflow.
  2. Select the PR's repository branch from the branch dropdown.
  3. Click Run workflow. Normal CI runs first; gpu-test starts only if that succeeds.

GitHub limits manual workflow dispatch to users with repository write access, but workflow YAML has no reliable “repository owner” role predicate. For owner-only enforcement, set the owner as a required reviewer on the GPU CI environment. A write-capable collaborator could dispatch the workflow, but only the configured owner can approve access to the environment and release VASTAI_API_KEY.

This replaces the earlier comment's statement that GPU tests run automatically on same-repository PRs; they no longer do.

@tetsuo-cpp tetsuo-cpp changed the title Run GPU tests in CI via Vast.ai ci: run optional GPU tests via Vast.ai Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant