Skip to content

Add Slurm + Apptainer support with CUDA 12.8 profile - #181

Merged
AbdelStark merged 4 commits into
mainfrom
cursor/slurm-apptainer-support-57f8
Nov 3, 2026
Merged

AbdelStark merged 4 commits into
mainfrom
cursor/slurm-apptainer-support-57f8

Conversation

@AbdelStark

@AbdelStark AbdelStark commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

Enables Hallmark's GPU workloads (TinyLlama LLM prover via SASH and classifier proofs via STWO) to run on generic Slurm + Apptainer clusters, in addition to the existing Hugging Face Jobs path.

Implementation

Container Profiles

Two Apptainer definition files with pinned base images:

  1. CUDA 12.9.1 (slurm/apptainer-cuda12.9.def, profile cuda12.9):

    • Base: nvidia/cuda@sha256:e542739fcaa4f45da5add8c4cf5769783a61628b2518304a5bbe4ace468b8c8f
    • CUDA version: 12.9.1 (per image config: CUDA_VERSION=12.9.1, NV_CUDA_CUDART_VERSION=12.9.79-1)
    • Driver: NVIDIA_REQUIRE_CUDA=cuda>=12.9 with datacenter branches 535, 550, 560, 565, 570
    • Validated with 580-branch drivers on HF Jobs
    • Compatible with existing HF Jobs evidence and build identity
    • On 570-branch drivers: Runs via CUDA forward compatibility (cuda-compat libraries), not natively
    • Caveat: Forward compatibility unverified inside TDX confidential VMs
  2. CUDA 12.8.1 (slurm/apptainer-cuda12.8.def, profile cuda12.8):

    • Base: nvidia/cuda@sha256:24c8e3581ea6330038b0d374920721983312627f8adbfcf390bdb4b399d280ed
    • CUDA version: 12.8.1
    • Driver: 570-branch or later (native support)
    • Recommended for clusters with 570-branch drivers, including TDX confidential VMs
    • New build identity (does not match HF evidence)

Both include: Rust nightly-2025-07-14, rustup 1.29.1, uv 0.9.18, Python 3.12, CUDA toolkit.

Driver 570-Branch Choice

On clusters with 570-branch drivers:

  • cuda12.8 (recommended): Native CUDA 12.8 support, new build identity, simpler deployment
  • cuda12.9 (alternative): Via forward compatibility, preserves HF build identity, requires cuda-compat setup, unverified in TDX

Unless preserving exact HF build identity is required, use cuda12.8 on 570-branch drivers.

Job Templates

Three sbatch templates with parameterized resources:

  • sbatch-cpu-smoke.sh: CPU smoke test (validate-data, check-conformance, circle-build)
  • sbatch-classifier-gpu.sh: Classifier GPU proofs (9 cases × 2 backends × 3 iterations)
  • sbatch-llm-gpu.sh: LLM GPU proofs (23 segments × 2 backends)

All templates support:

  • Configurable partition, reservation, nodelist, constraint
  • No hardcoded cluster names or paths
  • Run records with commit, image digest, CUDA version, driver, nvidia-smi, scontrol job info
  • Binary SHA-256 hashes

Launcher

slurm/hallmark-slurm.sh provides:

  • build-image: Build Apptainer SIF with profile selection (cuda12.8 or cuda12.9)
  • submit-cpu: Submit CPU smoke test
  • submit-classifier: Submit classifier GPU job
  • submit-llm: Submit LLM GPU job

Attestation Hook

slurm/attest-hook.sh (optional, off by default):

  • Computes manifest hash (inputs + binaries + proofs + verifier nonce)
  • Generates example commands:
    • TDX quote: sudo -n tdx-quote <64-hex-manifest-hash> (requires sudo, CPU-only job step)
    • GPU attestation: nvattest attest --device gpu --verifier local (nonce option depends on nvattest version)
  • Does NOT execute privileged tools automatically
  • Requires explicit opt-in and security review

Pass Criteria

Same as existing GPU benchmarks:

  • All segments and links verify
  • Zero CPU constraint fallback
  • Proof bytes match published proofs (for CUDA 12.9.1 profile)
  • Fresh proof-only verification passes

Documentation

What Can Be Verified Without GPU

  • Shell script syntax (all scripts pass bash -n)
  • Attestation hook script format
  • Definition file format
  • Documentation completeness
  • No hardcoded cluster details
  • Generic path structure
  • CUDA version from image config verified
  • Zero forbidden patterns (13.1, 13.x, cuda13, 570.211, apptainer-cuda12.def)

What Cannot Be Verified Without GPU

  • Container image build (requires Apptainer and network access)
  • CPU smoke test execution
  • GPU job execution
  • Proof verification on Slurm nodes
  • CUDA 12.8.1 numeric equivalence to CUDA 12.9.1
  • CUDA 12.9.1 forward compatibility on 570-branch drivers
  • Forward compatibility inside TDX confidential VMs

Files Changed

New Files (7)

  • slurm/apptainer-cuda12.9.def: CUDA 12.9.1 profile
  • slurm/apptainer-cuda12.8.def: CUDA 12.8.1 profile
  • slurm/sbatch-cpu-smoke.sh: CPU smoke test template
  • slurm/sbatch-classifier-gpu.sh: Classifier GPU job template
  • slurm/sbatch-llm-gpu.sh: LLM GPU job template
  • slurm/hallmark-slurm.sh: Launcher script
  • slurm/attest-hook.sh: Optional attestation hook

New Documentation (2)

  • docs/SLURM.md: Complete Slurm integration guide
  • docs/decisions/0092-slurm-apptainer-cuda12-profile.md: Decision record

Updated Files (3)

  • README.md: Added Slurm quickstart section
  • AGENTS.md: Added note about Slurm support
  • CHANGELOG.md: Added entry for this feature

Versioning

No version bump required. Python package (0.1.0) and Rust crate (0.1.0) remain unchanged. This adds an optional deployment path without modifying the core proof relation, protocol, or existing evidence. Version bumps in this repository track protocol/evidence changes (per decisions 0088, 0089), not deployment infrastructure additions.

CUDA Version Information

The existing pinned HF image is CUDA 12.9.1:

  • Image: nvidia/cuda@sha256:e542739fcaa4f45da5add8c4cf5769783a61628b2518304a5bbe4ace468b8c8f
  • Source: Docker Hub image config (verified via web lookup)
  • Config: CUDA_VERSION=12.9.1, NV_CUDA_CUDART_VERSION=12.9.79-1
  • Driver requirement: NVIDIA_REQUIRE_CUDA=cuda>=12.9 with datacenter branches 535, 550, 560, 565, 570
  • Validated: 580-branch drivers on HF Jobs (per REFERENCE-RUN.md)

Implementation Summary

Profile Names

  • cuda12.9: CUDA 12.9.1, existing HF baseline
  • cuda12.8: CUDA 12.8.1, native support for 570-branch drivers

Build Commands

# Existing HF baseline (CUDA 12.9.1)
./slurm/hallmark-slurm.sh build-image --profile cuda12.9 --output hallmark-cuda12.9.sif

# Recommended for 570-branch drivers (CUDA 12.8.1)
./slurm/hallmark-slurm.sh build-image --profile cuda12.8 --output hallmark-cuda12.8.sif

Job Submission

# CPU smoke test
./slurm/hallmark-slurm.sh submit-cpu --partition cpu --container hallmark-cuda12.9.sif

# Classifier GPU
./slurm/hallmark-slurm.sh submit-classifier --partition gpu --container hallmark-cuda12.9.sif \
  --prepared /data/$USER/prepared --native-pin <hex> --source-dir /data/$USER/source

# LLM GPU
./slurm/hallmark-slurm.sh submit-llm --partition gpu --container hallmark-cuda12.9.sif \
  --llm-run /data/$USER/llm-e2e-xyz --credit-balance 10.00

Attestation (Optional)

# Compute manifest hash and generate example commands
./slurm/attest-hook.sh <output-dir> <input-hashes> <binary-hashes> <proof-hashes> <nonce>

# Review generated commands in output-dir/attest-commands.sh
# Manually execute attestation tools with appropriate permissions

Related

  • Preserves existing HF Jobs path unchanged
  • Same worker entry points (scripts/run_llm_gpu.py, scripts/run_classifier_gpu.py)
  • Reuses research/probes/circle-gpu/build.sh for kernel builds
  • Compatible with existing evidence records (CUDA 12.9.1 profile)
  • Decision 0092 confirmed as next free decision number
Open in Web Open in Cursor 

cursoragent and others added 4 commits November 3, 2026 12:31
Enable Hallmark GPU workloads on generic Slurm clusters:
- Two CUDA profiles: cuda13 (CUDA 13.1, driver 580+) and cuda12 (CUDA 12.8.1, driver 570+)
- Apptainer definition files pinned by digest
- Sbatch templates for CPU smoke test, classifier GPU, and LLM GPU jobs
- Launcher script with parameterized partition/reservation/nodelist/constraint
- Run records with commit, image digest, CUDA version, driver, nvidia-smi, job info
- Optional attestation hooks (TDX quote, GPU attestation) documented but off by default

CUDA 12.8 profile enables TDX nodes with driver 570.x. CUDA 13.1 profile preserves
existing HF evidence. Same worker entry points and pass criteria as HF path.

Decision 0092, Slurm guide in docs/SLURM.md.

Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
1. Attestation: Replace incorrect CLI examples with actual tools
   - TDX: Use 'tdx-quote <64-hex>' (requires sudo, CPU-only job)
   - GPU: Use 'nvattest attest --device gpu --verifier local|remote'
   - Note nonce option depends on nvattest version
   - Create slurm/attest-hook.sh script (computes hash, generates commands, does not execute)
   - Update docs/SLURM.md and decision 0092

2. Remove site-specific examples
   - Replace 'driver_version: 570.211.01' with placeholders '<driver-version>'
   - Use generic values in example run records

3. Versioning: Resolve checklist item
   - No version bump required (0.1.0 unchanged)
   - Rationale: Adds deployment path, not protocol/evidence change
   - Decision 0092 acceptance checklist updated

4. CUDA version claim: Conservative update
   - Change 'CUDA 13.1' to 'CUDA 13.x' throughout
   - Add qualifier: 'per published benchmarks with driver 580.178.04'
   - Note: 'exact version unconfirmed from image manifest'
   - Update docs/SLURM.md, decision 0092, README, AGENTS, CHANGELOG

Decision 0092 is confirmed as next free number.

Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
1. CUDA version correction (existing image is 12.9.1, not 13.x):
   - Verified image config from Docker Hub
   - nvidia/cuda@sha256:e542739fcaa4f45da5add8c4cf5769783a61628b2518304a5bbe4ace468b8c8f
   - CUDA_VERSION=12.9.1, NV_CUDA_CUDART_VERSION=12.9.79-1
   - NVIDIA_REQUIRE_CUDA=cuda>=12.9 with datacenter branches 535,550,560,565,570

   Profile renaming:
   - cuda13 -> cuda12.9 (existing HF baseline, CUDA 12.9.1)
   - cuda12 -> cuda12.8 (new profile, CUDA 12.8.1)
   - Renamed slurm/apptainer-cuda13.def -> slurm/apptainer-cuda12.9.def

   Driver 570.x choice documented:
   - cuda12.8: Native CUDA 12.8 support (recommended, new build identity)
   - cuda12.9: Via forward compatibility (preserves HF identity, needs cuda-compat, unverified in TDX)

   Updated all references in:
   - docs/SLURM.md (profiles, examples, troubleshooting, version info)
   - docs/decisions/0092-slurm-apptainer-cuda12-profile.md
   - README.md, AGENTS.md, CHANGELOG.md
   - slurm/hallmark-slurm.sh (profile validation and examples)
   - slurm/apptainer-cuda12.9.def (labels and help text)

2. Delete SLURM_IMPLEMENTATION_SUMMARY.md from repo root:
   - Content belongs in PR description, not repository
   - PR description updated separately

Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
1. Fix decision record leftovers:
   - 'existing CUDA 13.1 profile' -> 'existing CUDA 12.9.1 profile' (2 instances)
   - 'for CUDA 13.1 profile' -> 'for CUDA 12.9.1 profile'
   - Remove specific driver versions (570.211.01, 580.178.04)
   - Use generic driver branch descriptions:
     - '570-branch drivers support CUDA 12.8 natively; CUDA 12.9 requires forward compatibility'
     - '580-branch drivers support CUDA 12.9 natively'

2. Rename for symmetry:
   - slurm/apptainer-cuda12.def -> slurm/apptainer-cuda12.8.def
   - Update reference in docs/SLURM.md

Verified: Zero hits for '13.1', '13.x', 'cuda13', '570.211', 'apptainer-cuda12.def' in:
- docs/
- slurm/
- README.md
- AGENTS.md
- CHANGELOG.md

Co-authored-by: abdel <AbdelStark@users.noreply.github.com>
@AbdelStark
AbdelStark merged commit 5233783 into main Nov 3, 2026
36 checks passed
@AbdelStark
AbdelStark deleted the cursor/slurm-apptainer-support-57f8 branch November 3, 2026 12:52
AbdelStark added a commit that referenced this pull request Nov 4, 2026
…resh agent context (#183)

Add `make compare-provers` and docs/PROVER-COMPARISON.md. The command re-times raw
cp_verify on the Mac (five calls per proof) and gathers the recorded SASH L1 and
FlashProve H200 figures with an evidence path for each. No paid compute; no
like-for-like prover claim.

Renumber the FlashProve decision 0092 -> 0093: #181 merged its Slurm decision as
0092 first. Decision 0093 now cites recorded verifier timings instead of
unrecorded ones. Route agents to the prover docs, record the Slurm profiles as
proposed and not run, and add both backends to the README, docs index, site
navigation, PERFORMANCE, ARCHITECTURE, STATE, ledger and changelog.

Co-authored-by: AbdelStark <AbdelStark@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants