Skip to content

Latest commit

 

History

History
150 lines (126 loc) · 15.9 KB

README.md

File metadata and controls

150 lines (126 loc) · 15.9 KB

Efficiency levers (research lane)

Tracking issue #43. Related: #7 (cost), #39 (prover speedup). Base: main 29907e5f (VERSION 0.16.0). Nothing here is a claim until merged and reviewed. Upstream figures are quoted from guide/performance.md, docs/figures.tsv (row ids in backquotes) and paper/paper.pdf section 7. All upstream timings: one H100 80 GB of an 8-GPU host, 24 CPUs, public 4-bit Maverick files unless marked synthetic, n = 3 runs.

1. Fact-check of the public cost teardown

Source: 0xOsprey, "FlashProve economics, raw vs proven inference", a GitHub gist whose id begins 52fd0803. The full id and revision were not recorded when this section was written. The gist is not in the author's public gist listing as of 2026-11-04, so the full URL cannot be given here. Headline: ~$0.017 per proven 1,024-token request, ~$15/Mtok proven vs ~$0.11 raw, ~165x all-in.

Legend. UP = upstream-measured, matches the repo. CONV = the author's own conversion or assumption (arithmetic checked). WRONG = does not match the repo or the arithmetic. NUANCE = number right, framing loses a caveat the repo states. HALF RIGHT / MOSTLY RIGHT = part of the claim matches the repo, the rest does not (the row says which). NOT FOUND = the cited source does not contain it. OUT OF SCOPE = not checkable from this repository.

Teardown number Verdict Evidence / correction
111.23 tok/s at batch 24 ("400k tok/hr") UP rate-24 111.23 ± 0.92, n = 3. 111.23 x 3600 = 400,428 tok/h.
17.4 / 76.3 / 123.3 tok/s at batch 1 / 8 / 64 UP rate-1, rate-8, rate-64.
Waits ~59 / ~107 / ~222 / ~536 s UP for 24 and 64, CONV for 1 and 8 latency-24 220.7 to 225.9, latency-64 533.3 to 537.9. 1 and 8 are seconds-per-request x batch (58.8; 13.41 x 8 = 107.3); no measured wait exists for them.
batch 64: 80,629 of 81,559 MiB; synthetic attempts at 64 all fail at a CUDA call UP gpu-mem-64, failed-batch, service-at-64-synthetic (0 of 320). Public weights at 64: 960 of 960 accepted. Cause of the synthetic failure is "not known" upstream: OOM is the teardown's inference, not a finding.
18.78 MB proof, 6.24 MB client download, 2.15 GB provider secret dir UP proof-size-mean, client-download, secret-dir-size.
Commit 38.7 min UP commit-time-drawn 38:43.77.
688k batches per leaf table UP batches-per-leaf-table 688,127.
cp_verify ~21 s (1 thread), ~17 s (8-24 threads), <= 3.53 GB UP verify-time 21.14, verify-time-8-threads 16.84, verify-time-24-threads 17.07, verify-memory 3.53 GB.
Lean verify ~24 min, <= 14.13 GB UP, not released programs lean-verify-time 1,414 s; marked † (not the released programs).
Middleware check ~94 s per batch of 24 UP, n = 1 serving-check-24, one run, synthetic, not a timed run.
"~280 GB host RAM" UP, rounded container-mem-no-cache 269.7 GB public / 278.3 GB synthetic; host-mem 257.1 GB (process).
93x (max load), 71.4x (same batch), 20.1x (vs decode) UP (calculated upstream) overhead-prefill-max, overhead-prefill-24, overhead-decode. Baseline: vLLM FP8 on 8x H100, separate days; upstream says baseline is a lower bound.
Unproven ~37.2M tok/GPU-h, ~$0.094/Mtok GPU-only CONV, arithmetic OK 111.23 x 93.0 x 3600 = 37.24M tok/h; $3.50 / 37.24 = $0.094/Mtok.
Proven ~$15/Mtok CONV, arithmetic OK given his inputs ($3.50 GPU + $3.00 host) / 0.4004 Mtok/h = $16.2/Mtok; GPU only $8.74. "~$15" is mildly low for his own $6.5/h.
$/Mtok ~$96 / ~$22 / ~$15 / ~$13.5 at batch 1 / 8 / 24 / 64 CONV, slightly low At $6.5/h: $103.7 / $23.7 / $16.2 / $14.6. His column looks computed at ~$6.0/h.
~$0.017 per request CONV, OK 9.206 s x $6.5/h = $0.0166.
~165x all-in CONV, depends on two assumptions 93 x (6.5 / 3.5) = 172.7x if the raw side pays no host share; with his $0.016/Mtok host share on the raw side, ~147x. Headline sits inside his own 100-170x range; it is not an upstream number.
Host ~$3/h for 24 vCPU + ~280 GB CONV, unsourced No upstream price. It is the largest single assumption after GPU $/h. Upstream ratios are "per GPU" and explicitly exclude CPUs and host memory (performance.md "Cost").
Middleware ~23 s of 1 CPU per request CONV 94.0 s x 6 parallel checks / 24 = 23.5 CPU-s. OK.
"Proving costs ~the same whether 1 or ~360 tokens are checked" / "~26x per attested token" NUANCE Upstream: 108.67 vs 108.73 tok/s with every reply token checked vs last token only (rate-replies-all, rate-replies-control), "the difference shows nothing". Stored replies of 31-500 tokens, chosen to meet limits. The 26x is his division, not upstream.
6.81% integer vs FP16 top-token divergence; 0/96 exact match UP integer-vs-float, gap-honest-replies-exact-match. Python implementation, not released programs.
0.58-1.46 s/token through the integer-exact vLLM plug-in UP, † performance.md "Proving, not serving", 48 of 48 short replies, not released programs.
Run-to-run spread "up to 9%" UP, † performance.md "How far the figures repeat".
"6 passes over ~200 GB of weights: 4 waves + 2 walks; ~37 s/pass" MOSTLY RIGHT on structure, WRONG on cost split 4 waves = 24 / -in-step 6 (measure_rate.py default); 2 walks = hf_side_walks() "one pass for u, one for all opened columns" (prover/hash_commitment/hf_keeper.cu:362). But passes are not equal cost: see section 2. Weight files are 232.13 GB (weight-files), not ~200 GB.
"Binding constraint: the 80 GB card holds ~29% of ~280 GB working set; weights stream from pinned host RAM every pass" HALF RIGHT Streaming from pinned host memory is real (prover/prover/weight_stream.cuh header: ZeRO-Inference / FlexGen style). "Binding" is not supported: upstream numbers decompose to ~58% per-request compute (section 2). 80/280 is his ratio.
"Main gain is single-request latency" attributed to "report section 7"; "meaningful optimizations we did not incorporate" quote NOT FOUND pdftotext paper/paper.pdf has no "incorporat" string and no H200/B200 mention; section 7 is Evaluation. The quote may come from a technical report the repo calls "not public". Treat as unsourced.
"Opening more columns past the mask budget (44,040,191) fails ZK: this IS the batch-size ceiling" NUANCE The mask budget limits columns per leaf table over its life (688,127 batches); the per-batch ceiling in code is SLOTS_MOST = 64 (prover/prover/slot.h) and the parameter file's slots.
Anthropic launch-post corroboration, VerInf/Attestable/DeepProve comparison OUT OF SCOPE Not checkable from this repo. paper Table 8 lists the same systems "without ratios, because the machines, models and proved statements differ"; the gist computes ratios anyway.

Net: every upstream-measured number he uses is quoted correctly. The dollar figures are his conversions on two unsourced prices ($3.50 GPU, $3.00 host); his batch cost row is computed at about $6.0/h, not the $6.5/h he states. The weakest part is the bottleneck story, below.

2. Bottleneck anatomy

Claim under test (teardown): GPU memory is the binding constraint.

What the code says:

  • Weights live in pinned host memory and are copied to a few GPU buffers ("rooms") group by group (prover/prover/weight_stream.cuh, header comment). -memory lean = late-on-keys,tensor-rooms=10 (prover/prover/arguments.h presets()); the service default is lean,tensor-rooms=8 (driver/measure_rate.py, integrations/vllm/REFERENCE.md).
  • A batch of B requests runs ceil(B / in-step) batch passes ("waves"), each one read of the weights, then the opening's 2 walks (hf_keeper.cu:362). -in-step takes 1 to 16 (arguments.h); "whether more than six requests in step fit is not known" (integrations/vllm/REFERENCE.md).
  • -overlap / -overlap-two fold a batch's walks into the next batch's waves, at the cost of a second / third weight-table keeper and engine in GPU memory; refused on a model's files (synthetic only).
  • One code comment gives a single untimed run at full size, -in-step 6: batch passes 73-90.5 s, the opening's passes 34.6-38.2 s (weight_stream.cuh, comment above load()).
  • Build targets sm_90a only (prover/build.sh line 617, -arch=sm_90a). B200 (sm_100) cannot run these binaries without a port. H200 is sm_90 and runs them.

What upstream numbers say. Fit batch time t(B) = a + bceil(B/6) + cB to the five measured batch sizes (B = 1, 8, 24, 48, 64; 48 is † not released programs; t = B*1024/rate):

term fitted meaning
a 38.2 s per-batch fixed: the opening's 2 walks + batch end
b 14.0 s per wave: one weight stream not hidden behind compute
c 5.32 s per request: that request's sub-proofs over its 1,024 tokens

Fit residuals are under 1.5 s at every B (B = 1: 57.5 vs 58.8). At B = 24: 38 s fixed (17%), 56 s waves (25%), 128 s per-request (58%). Read: three parameters on five points, two of them partly collinear; this is a decomposition hypothesis, not a measurement. (Corrected 2026-11-10: an earlier version said the GPU run in section 5 measures this directly with -measure. It does not; no -measure breakdown was taken, so the fitted fractions remain a hypothesis.)

Consequence: if per-request compute is ~58% of a batch, more GPU memory alone cannot give more than ~1.7x at batch 24, and only if it removes all wave and walk overhead. Memory is a real constraint on batch size (64 fails on synthetic weights) and on in-step, but not obviously the binding constraint on throughput.

3. Ranked levers

# Lever Expected gain on tok/s (batch 24 basis) Evidence Cost / risk
1 Bigger GPU memory used for fewer waves: H200 141 GB with -in-step 12..16 and batch 48 +15 to +35% predicted (fit: B = 48, in-step 16 -> 3 waves instead of 8 -> ~147 tok/s vs 121 at in-step 6) fit above; rate-48-steady 120.94 †; in-step cap 16 in arguments.h needs H200; at the Vast quotes used here one H200 cost $5.54/h against $0.61 to $2.41 per H100 GPU (one GPU of a 2-GPU box at $1.22/h or $4.82/h; rental 1 and 2 offers, experiments/20261104-43-h200-memory-lever/offers.json), 2.3x to 9x per GPU, so $/Mtok is the real test; latency rises with B
2 Batch size alone (24 -> 48/64) on 80 GB +8.7% (48), +10.9% (64) measured rate-48-steady †, rate-64 64 unreliable on 80 GB; latency 2.4x
3 Same config on H200 (HBM 4.8 vs 3.35 TB/s, same SM count, same PCIe Gen5 host link) 0 to +10% predicted no prover code adapts room count to free memory automatically; compute dominates pure hardware price question
4 -overlap / -overlap-two up to the 38 s fixed term (~17%) if walks hide fully arguments.h; synthetic-only (refused on a model's files) more GPU memory; not usable on real weights today
5 Fill the window / check whole replies same tok/s, more attested tokens per proof rate-replies-all vs control coverage choice, not speed
6 NUMA binding, huge pages commit 38% (wrong node), prove ~2% performance.md last section (†, untimed) operational, already practice
- CPU micro-tweaks skip PR #40: 6.5%

B200: excluded from the experiment, the binaries are sm_90a only. 2x H100: the prover pins device 0 (prover/prover/inputs.cuh cudaSetDevice(0)); multi-GPU is a code change, not a knob.

4. GPU experiment plan

See experiments/20261104-43-h200-memory-lever/PREREG.md (written before any GPU code).

5. Results

Full numbers, verdict and files: experiments/20261104-43-h200-memory-lever/RESULTS.md (run record, as committed) and RESULTS-ADDENDUM-2026-11-10.md in the same directory (current reading; it wins where the two differ). Measured, unreplicated, synthetic weights, n = 1 session (4 batch intervals) per arm. This section was rewritten 2026-11-10.

  • Verdict: lever not supported; effect not identifiable in this design. Rental 2: H200 in-step 12 (TB1) 146.09 tok/s against H100 in-step 6 (CA1) 93.91 tok/s, each 24,576 / the session's median batch interval, which is 1.556x with disjoint per-interval ranges. That clears the 1.15x bar as computed. But the two arms ran on two different hosts, one session each, and the H100 control is off baseline (14% below upstream's 109.09; its full-size commit took 2.11x as long as the H200 box's). GPU memory and host speed are confounded, so this design cannot say how much of the 1.556 either explains. No supported throughput improvement; this is not evidence of no effect either.
  • Cost: FAIL. At the rented prices T-B is $10.53/Mtok against C-A $7.13/Mtok.
  • GPU memory. Observed: in-step 12 at batch 24 ran on the H200 (120 of 120 proofs accepted) with a sampled peak of 87,237 MiB (sampling stopped at 889.5 s, so a later peak is not seen); the H100's total is 81,559 MiB. Inferred, not measured: that configuration probably does not fit an 80 GB H100. No committed file shows it run or fail on an H100; rental 1's CB2 start-up failure (p258 end 1 cuda-call) is run-log text without a surviving file and is not shown to be an out-of-memory error.
  • Rental 1 data after 02:33 UTC is lost (never pulled before the 04:45 UTC backstop destroyed the boxes). CA1, TA1 and CB2 survive only as the run-log text below, without files; no conclusion here rests on them.
  • Limitations: both rental-2 session logs carry measure_rate.py's clock-disagreement warning (> 2 s between clock readings at a release; 3 lines in TB1, 10 in CA1); see the addendum. No receipt file is committed for either rental.
  • Next test that could separate the effects: both in-step settings on one H200 host (in-step 6 vs 12, AB/BA, >= 5 sessions each), which removes the host confound entirely.

Run log

Written during rental 1 (2026-11-04). Historical text: none of rental 1's session, report or timing files survives, and the readings in this log are superseded by the results above (bracketed notes added 2026-11-10).

  • 2026-11-04 01:22 UTC: two instances created (control: 1 GPU of a 2x H100 SXM 80 GB box, $1.219/h; treatment: 1x H200 SXM 141 GB, $5.54/h). Teardown timers armed before create. Both boxes started the pinned build at ~01:31 UTC.
  • 01:42-01:43 UTC: pinned build + examples/quickstart-gpu.sh passed on both boxes (cp_verify YES on the small proof). ~17 min each.
  • Early commit observation (n = 1 per box, the commitment's anchor table, from cp_commit's own timing lines): H200 splits the table's tree into 16 parts of 2^28 positions, 452.7 s; H100 into 32 parts of 2^27, 576.3 s (-21%). Inference, not checked in code: the committer appears to size its parts to free GPU memory (16 x 2^28 vs 32 x 2^27). It concerns commit, not prove.
  • Full-size synthetic commitment (cp_commit -weights made-up-blocks, target_full_f2, -M 1), n = 1 per box, wall from stage log: H200 1,698 s (01:43:26 to 02:11:44), H100 1,871 s (01:42:32 to 02:13:43): H200 9% faster. Upstream commit-time-synthetic 2,312.68 s (other hosts/CPUs). Phase anchor-table 461.4 vs 585.1 s; key-file 240.6 vs 242.6 s. Sessions started 02:11 (H200) and 02:13 (H100).
  • 02:33 UTC, first sessions (n = 1 each, 4 intervals each, every proof checked by cp_verify):
    • H100 CA1 (24/6): STEADY 146.67 tok/s, intervals 168.18 / 167.53 / 167.24 / 167.27 s, GPU peak 74,641 MiB, cp_prove RSS 265.5 GB.
    • H200 TA1 (24/6): STEADY 142.97 tok/s, intervals 171.13 / 172.28 / 172.16 / 172.01 s, GPU peak 74,673 MiB.
    • Same config on H200 is not faster (-2.5%), and uses the same ~74.7 GB: the prover does not use the extra memory by itself. [2026-11-10: one unbound session per box on two hosts, files lost; not established.]
    • Both are ~34% above upstream's 109.09 synthetic (24 CPUs bound). These boxes run unbound with a 30.7-core (H100) / 23-core (H200) cgroup quota on 128/144-thread hosts. Host CPU is a confound larger than the GPU difference. [2026-11-10: a hypothesis from one session per box, not established.]
    • H100 CB2 (24/12): p258 end 1 cuda-call at start-up, no proof. in-step 12 does not fit on 80 GB. [2026-11-10: the failure is run-log text without a surviving file and is not shown to be out of memory; the fit statement is an inference, see section 5.]