Repository navigation
Add Slurm + Apptainer support with CUDA 12.8 profile - #181
Merged
Merged
Conversation
Enable Hallmark GPU workloads on generic Slurm clusters: - Two CUDA profiles: cuda13 (CUDA 13.1, driver 580+) and cuda12 (CUDA 12.8.1, driver 570+) - Apptainer definition files pinned by digest - Sbatch templates for CPU smoke test, classifier GPU, and LLM GPU jobs - Launcher script with parameterized partition/reservation/nodelist/constraint - Run records with commit, image digest, CUDA version, driver, nvidia-smi, job info - Optional attestation hooks (TDX quote, GPU attestation) documented but off by default CUDA 12.8 profile enables TDX nodes with driver 570.x. CUDA 13.1 profile preserves existing HF evidence. Same worker entry points and pass criteria as HF path. Decision 0092, Slurm guide in docs/SLURM.md. Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
1. Attestation: Replace incorrect CLI examples with actual tools - TDX: Use 'tdx-quote <64-hex>' (requires sudo, CPU-only job) - GPU: Use 'nvattest attest --device gpu --verifier local|remote' - Note nonce option depends on nvattest version - Create slurm/attest-hook.sh script (computes hash, generates commands, does not execute) - Update docs/SLURM.md and decision 0092 2. Remove site-specific examples - Replace 'driver_version: 570.211.01' with placeholders '<driver-version>' - Use generic values in example run records 3. Versioning: Resolve checklist item - No version bump required (0.1.0 unchanged) - Rationale: Adds deployment path, not protocol/evidence change - Decision 0092 acceptance checklist updated 4. CUDA version claim: Conservative update - Change 'CUDA 13.1' to 'CUDA 13.x' throughout - Add qualifier: 'per published benchmarks with driver 580.178.04' - Note: 'exact version unconfirmed from image manifest' - Update docs/SLURM.md, decision 0092, README, AGENTS, CHANGELOG Decision 0092 is confirmed as next free number. Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
1. CUDA version correction (existing image is 12.9.1, not 13.x): - Verified image config from Docker Hub - nvidia/cuda@sha256:e542739fcaa4f45da5add8c4cf5769783a61628b2518304a5bbe4ace468b8c8f - CUDA_VERSION=12.9.1, NV_CUDA_CUDART_VERSION=12.9.79-1 - NVIDIA_REQUIRE_CUDA=cuda>=12.9 with datacenter branches 535,550,560,565,570 Profile renaming: - cuda13 -> cuda12.9 (existing HF baseline, CUDA 12.9.1) - cuda12 -> cuda12.8 (new profile, CUDA 12.8.1) - Renamed slurm/apptainer-cuda13.def -> slurm/apptainer-cuda12.9.def Driver 570.x choice documented: - cuda12.8: Native CUDA 12.8 support (recommended, new build identity) - cuda12.9: Via forward compatibility (preserves HF identity, needs cuda-compat, unverified in TDX) Updated all references in: - docs/SLURM.md (profiles, examples, troubleshooting, version info) - docs/decisions/0092-slurm-apptainer-cuda12-profile.md - README.md, AGENTS.md, CHANGELOG.md - slurm/hallmark-slurm.sh (profile validation and examples) - slurm/apptainer-cuda12.9.def (labels and help text) 2. Delete SLURM_IMPLEMENTATION_SUMMARY.md from repo root: - Content belongs in PR description, not repository - PR description updated separately Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
1. Fix decision record leftovers:
- 'existing CUDA 13.1 profile' -> 'existing CUDA 12.9.1 profile' (2 instances)
- 'for CUDA 13.1 profile' -> 'for CUDA 12.9.1 profile'
- Remove specific driver versions (570.211.01, 580.178.04)
- Use generic driver branch descriptions:
- '570-branch drivers support CUDA 12.8 natively; CUDA 12.9 requires forward compatibility'
- '580-branch drivers support CUDA 12.9 natively'
2. Rename for symmetry:
- slurm/apptainer-cuda12.def -> slurm/apptainer-cuda12.8.def
- Update reference in docs/SLURM.md
Verified: Zero hits for '13.1', '13.x', 'cuda13', '570.211', 'apptainer-cuda12.def' in:
- docs/
- slurm/
- README.md
- AGENTS.md
- CHANGELOG.md
Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
AbdelStark
added a commit
that referenced
this pull request
Nov 4, 2026
…resh agent context (#183) Add `make compare-provers` and docs/PROVER-COMPARISON.md. The command re-times raw cp_verify on the Mac (five calls per proof) and gathers the recorded SASH L1 and FlashProve H200 figures with an evidence path for each. No paid compute; no like-for-like prover claim. Renumber the FlashProve decision 0092 -> 0093: #181 merged its Slurm decision as 0092 first. Decision 0093 now cites recorded verifier timings instead of unrecorded ones. Route agents to the prover docs, record the Slurm profiles as proposed and not run, and add both backends to the README, docs index, site navigation, PERFORMANCE, ARCHITECTURE, STATE, ledger and changelog. Co-authored-by: AbdelStark <AbdelStark@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enables Hallmark's GPU workloads (TinyLlama LLM prover via SASH and classifier proofs via STWO) to run on generic Slurm + Apptainer clusters, in addition to the existing Hugging Face Jobs path.
Implementation
Container Profiles
Two Apptainer definition files with pinned base images:
CUDA 12.9.1 (
slurm/apptainer-cuda12.9.def, profilecuda12.9):nvidia/cuda@sha256:e542739fcaa4f45da5add8c4cf5769783a61628b2518304a5bbe4ace468b8c8fCUDA_VERSION=12.9.1,NV_CUDA_CUDART_VERSION=12.9.79-1)NVIDIA_REQUIRE_CUDA=cuda>=12.9with datacenter branches 535, 550, 560, 565, 570CUDA 12.8.1 (
slurm/apptainer-cuda12.8.def, profilecuda12.8):nvidia/cuda@sha256:24c8e3581ea6330038b0d374920721983312627f8adbfcf390bdb4b399d280edBoth include: Rust nightly-2025-07-14, rustup 1.29.1, uv 0.9.18, Python 3.12, CUDA toolkit.
Driver 570-Branch Choice
On clusters with 570-branch drivers:
Unless preserving exact HF build identity is required, use cuda12.8 on 570-branch drivers.
Job Templates
Three sbatch templates with parameterized resources:
sbatch-cpu-smoke.sh: CPU smoke test (validate-data, check-conformance, circle-build)sbatch-classifier-gpu.sh: Classifier GPU proofs (9 cases × 2 backends × 3 iterations)sbatch-llm-gpu.sh: LLM GPU proofs (23 segments × 2 backends)All templates support:
Launcher
slurm/hallmark-slurm.shprovides:build-image: Build Apptainer SIF with profile selection (cuda12.8 or cuda12.9)submit-cpu: Submit CPU smoke testsubmit-classifier: Submit classifier GPU jobsubmit-llm: Submit LLM GPU jobAttestation Hook
slurm/attest-hook.sh(optional, off by default):sudo -n tdx-quote <64-hex-manifest-hash>(requires sudo, CPU-only job step)nvattest attest --device gpu --verifier local(nonce option depends on nvattest version)Pass Criteria
Same as existing GPU benchmarks:
Documentation
What Can Be Verified Without GPU
bash -n)What Cannot Be Verified Without GPU
Files Changed
New Files (7)
slurm/apptainer-cuda12.9.def: CUDA 12.9.1 profileslurm/apptainer-cuda12.8.def: CUDA 12.8.1 profileslurm/sbatch-cpu-smoke.sh: CPU smoke test templateslurm/sbatch-classifier-gpu.sh: Classifier GPU job templateslurm/sbatch-llm-gpu.sh: LLM GPU job templateslurm/hallmark-slurm.sh: Launcher scriptslurm/attest-hook.sh: Optional attestation hookNew Documentation (2)
docs/SLURM.md: Complete Slurm integration guidedocs/decisions/0092-slurm-apptainer-cuda12-profile.md: Decision recordUpdated Files (3)
README.md: Added Slurm quickstart sectionAGENTS.md: Added note about Slurm supportCHANGELOG.md: Added entry for this featureVersioning
No version bump required. Python package (0.1.0) and Rust crate (0.1.0) remain unchanged. This adds an optional deployment path without modifying the core proof relation, protocol, or existing evidence. Version bumps in this repository track protocol/evidence changes (per decisions 0088, 0089), not deployment infrastructure additions.
CUDA Version Information
The existing pinned HF image is CUDA 12.9.1:
nvidia/cuda@sha256:e542739fcaa4f45da5add8c4cf5769783a61628b2518304a5bbe4ace468b8c8fCUDA_VERSION=12.9.1,NV_CUDA_CUDART_VERSION=12.9.79-1NVIDIA_REQUIRE_CUDA=cuda>=12.9with datacenter branches 535, 550, 560, 565, 570Implementation Summary
Profile Names
cuda12.9: CUDA 12.9.1, existing HF baselinecuda12.8: CUDA 12.8.1, native support for 570-branch driversBuild Commands
Job Submission
Attestation (Optional)
Related
scripts/run_llm_gpu.py,scripts/run_classifier_gpu.py)research/probes/circle-gpu/build.shfor kernel builds