An open benchmark for provable inference.
The same models, the same reference hardware, the same timing rules,
and a claim card beside every number saying what the proof establishes.
Results · Learn · Specification · RFCs · PRD · Add a system · Glossary
- Why this exists
- What gets measured
- Anatomy of a number
- The claim card
- Timing scopes
- Negative controls
- How a run works
- Level 1, release v1.0
- Quick start
- Repository layout
- Principles and non-goals
- Contributing
- Licence
Several open systems can now prove that a language model produced a given output. Each publishes its own numbers, and no two of them can be read side by side, because four things differ at once:
| Differs | Example of the gap | What this benchmark fixes |
|---|---|---|
| Model | an 8B dense model vs a mixture-of-experts model at another precision | pinned workloads: model by public source and content hash, prompt sets by digest |
| Hardware | each author's own machine and driver | pinned reference profiles with a machine record and CPU binding |
| Timing rule | "proving time" that does or does not include start-up, commitment or batching | named timing scopes, measured by the harness from outside the prover |
| Claim | one proof binds the weights, another samples, a third proves a single layer | a claim card per configuration, copied from the system's own documentation |
The first three can be removed. The fourth cannot: two systems that prove different statements are not faster or slower than each other, they are doing different work. So this project removes the differences it can and labels the one it cannot. It is in the spirit of Ethproofs for zkVM provers, with one addition that inference needs: the claim card.
Important
This project runs provers and ranks them inside one model's rung by one measured figure, the cost of a proved reply as a multiple of serving it, among the systems that pass six checks we run ourselves (RFC-0018). It does not build or certify them, and it ranks nothing across models.
Each system runs at a full commit hash, with no setting its documentation does not name, behind a five-subcommand adapter contract. The v1.0 runs are hosted: their manifests record the provider's network, and the verify trace records the files the verifier opens, not its connections.
info → setup → commit → prove → verify
The harness times every call and every proof release from outside the container (RFC-0011), samples GPU and host resources through NVML and cgroups, runs the same tampering controls against every verifier (RFC-0008), and recomputes every derived figure in CI from the raw records (RFC-0004).
Each system is also set beside reference rows (RFC-0015): the same model served by a production engine (vLLM, publisher's release precision, defaults only) on the same GPU generation. The overhead of proving is read against that row, in GPU-seconds per token, with its scope on the label.
No number exists in this repository without its labels (one quantity/1 type, RFC-0012). This is one, verbatim, from a stored v0.1 run:
Three vocabularies label it. Each is normative in the RFC named beside it.
| Grade: how it was obtained RFC-0004 | Evidence tier: who stands behind it RFC-0007 | Wording: what a proof does Glossary | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Every tier the harness records is |
Never "provably safe". |
Derived numbers are computed, never typed. CI recomputes every one from its inputs, bit for bit, and fails on any difference.
A claim card (RFC-0002) is the machine-checkable statement, per system configuration, of what an accepted proof establishes: the relation, the security notion and level, the assumptions, which edges of the computation are bound, and the residual, what a cheating prover can still do while passing verification.
Every field is copied from the system's own documentation at the pinned commit, with a line-anchored link, or written not stated where the source is silent. Nothing is inferred. An excerpt from systems/sash/claim.yaml:
upstream: {repo: "sg-ai-safety-hub/zk-inference-prover", commit: "4f41667297430a598165c4a8604258fe37d952ae"}
claim_class: "greedy-continuation/exact-match/committed-weights"
security:
notion:
kind: "not_stated"
level: >-
100 conjectured bits for each segment proof, by stwo's own conjectured-soundness accounting
[...] A level for the whole grid of segments is not stated.
zero_knowledge: {claimed: "not stated", hidden: "not stated", not_hidden: "not stated", status: "not_stated"}A system enters the ranking by passing six checks we run ourselves (RFC-0018):
| Check | Passes when |
|---|---|
| Run | three sessions, each from a cold start on its own prompts, built from the pinned public commit, each returned an accepted proof for every request |
| Model | a proof made after changing one committed weight was never accepted, and an honest proof was refused against another commitment |
| Prompt | a proof was refused for a prompt with one token changed |
| Reply | proofs cover all 512 reply tokens, and a changed token at the first, the last and an interior position was refused |
| Verify without weights | every verdict came from a verify call with no GPU and no model file, under a file-open trace |
| Fidelity | the committed copy's replies match the publisher's original weights for more leading tokens than the bar's: a floor, not a measure of closeness |
None of the checks attempts a forged proof: soundness and weight secrecy are not tested.
The glyph then shows how far a system that ran is from a perfect prover (RFC-0018). Five petals, each at one step: Met, Mostly met, Partly met, Not met, Not stated, Failed (our check or our build of the publisher's proof contradicted it) or Not decided. The steps are drawn as inks; no printed word names an ink:
| Petal | Protects | Met means |
|---|---|---|
| Integrity (soundness) | the customer | a bound on a false reply being accepted, proved, that covers what the row runs, with every exponent in the stated bound 128 or more (a term 2^-t or q/2^t has exponent t) |
| Confidentiality (zero knowledge of the weights) | the provider | zero knowledge proved for the configuration the row runs, the programs stated to implement it, and a published list of what stays visible with nothing that depends on the weights |
| Fidelity (quantization error, in matched leading tokens) | the customer | the proved replies are shown above halfway between a second run of the publisher's model and its 8-bit copy, in a fidelity run of one request at a time |
| Formally verified (machine-checked soundness proof) | the customer | a machine-checked soundness theorem covers the verifier code the row runs, with its trusted base listed and no input-reading code outside the proof, and a build of it at the pin, with the toolchain the publisher pins, is recorded |
| Trust assumptions (trusted setup or party) | the customer | soundness rests on cryptography alone, and no setup ever produced a secret |
Integrity, Confidentiality, Formally verified and Trust assumptions are computed from facts a reviewer copies, each with its quote, from the publisher's documents at the pin (systems/<name>/properties.yaml); no state is typed by hand. A value that is the reader's reading rather than the publisher's words carries a note; the system page marks the petal "read as" and grades it Mostly met at most. Fidelity is measured by our fidelity run. A failed check sets Failed on the petals it contradicts. Each system page prints, under each petal, its next step: what the row lacks for the step above, and who acts. Nothing adds the petals up, and the glyph never moves a position: the order is cost among the systems that passed.
Every time and every rate names exactly one scope (RFC-0004 INV-SCOPE-1). Boundaries are harness clock events (time.monotonic_ns()); a proof's release time is the instant the harness observes its atomic rename into /io/proofs/.
| Scope | Starts at | Ends at | Tokens attributed |
|---|---|---|---|
commit |
commit call start |
commit call exit |
none |
prove.session |
prove call start |
release of the last proof | all proofs |
prove.steady |
last release in the first batch | release of the last proof | requests after the first batch |
whole_wall |
commit call start |
release of the last proof | all proofs |
verify |
verify call start, 8-CPU verifier |
that call's exit | the proof's tokens |
verify.single_thread |
same, 1-CPU verifier | that call's exit | the proof's tokens |
prove.steady excludes start-up; prove.session includes it. A system that releases no proof before the whole request is proved has no steady scope, and its headline says so on the label.
A benchmark that only times honest runs cannot tell a verifier from one that accepts everything. Every verifier faces the same tampering controls (RFC-0008), applied by the harness, never by the system:
| Id | Tampering | Expected |
|---|---|---|
| C1 | change the claimed output token | reject |
| C2 | change one prompt token in the request | reject |
| C3 | verify against the commitment of perturbed weights | reject |
| C4 | replay a valid proof for a different request | reject |
| C5 | truncate the largest part the verifier reads | reject or invalid_call |
| C6 | flip one bit at a seeded offset in a part the verifier reads, 5 seeds | reject or invalid_call, each seed |
| C7 | swap the proofs of two requests in one batch | reject |
| C8 | drop one part the verifier reads | reject or invalid_call |
| P1 | commit W, prove with perturbed W′ | prover_failed or reject |
| P2 | prove at a precision other than the committed one | prover_failed or reject |
Each control has an honest twin: the untouched proof, verified in the same batch with the same image. If the twin is not accepted, the control is void, never counted as a rejection. Switches that weaken a system's statement (for example SASH's --unsound-skip-*) run only in control jobs, where their documented acceptance is the expected outcome, and never produce a measured row.
flowchart LR
W["Workload<br/>model hash · prompt digest"] --> H
P["Reference profile<br/>GPU · CPUs · driver"] --> H
H["iproofs run<br/>binds GPU and CPUs,<br/>checks the host"] --> A
subgraph A["Adapter container · pinned commit · no network"]
direction LR
s[setup] --> c[commit] --> p[prove] --> v[verify]
end
A --> R["Run directory<br/>run.json · manifest · digests · logs"]
R --> D["iproofs derive<br/>named formulas"]
R --> C["iproofs controls<br/>C1–C8, P1–P2"]
D --> F["iproofs release freeze<br/>index.json"]
C --> F
K["Claim cards"] --> F
F --> S["iproofs site build<br/>inferenceproofs.ai"]
The commitment the verifier checks is taken from the harness's own record of the commit step, never from files shipped with a proof. Proof bytes are not in the repository and are not public yet: each run directory names them by sha256 in digests.txt, and the bytes sit in a private bucket (RFC-0009, RFC-0014). docs/REPRODUCE.md says what can be checked without them.
Level 1 (RFC-0018, levels/L1.yaml, frozen 2026-11-11) is a ladder of three open-weights models: Llama 4 Maverick, Llama 3.1 8B and GPT-2. Release v1.0's board, releases/v1.0/board.json, is computed from the run records in results/v1.0/ by iproofs release board v1.0, on NVIDIA H200 141 GB hosted profiles. Every figure in it is at evidence tier reported: our runs only, not rerun by anyone else. docs/REPRODUCE.md rebuilds it.
| System | Pinned upstream | Rung | Claim card |
|---|---|---|---|
| FlashProve | anthropics/FlashProve@9e2bdd8 (v0.16.0) |
Llama 4 Maverick (l1-maverick-p512-o512) |
exact-match-cp-verify |
| SASH zk-inference-prover | commit 4f41667 of pull request #36, written by this project's owner and not merged upstream; upstream eb95bfa refuses one of the level's prompts |
Llama 3.1 8B, W8A8 (l1-l31-8b-p512-o512) |
sound-defaults |
| DeepProve | Lagrange-Labs/deep-prove@9d1a53e. prove is wired, verify refuses: the upstream verifier takes its verifier key only from inside the proof file, and no documented step makes that key from the weights alone, so no session can complete (systems/deepprove/README.md) |
GPT-2 (l1-gpt2-p512-o512) |
none |
| Production serving (reference) | vLLM | each rung's model at the publisher's release precision | not a proof |
No two systems share a rung, so nothing on the board is ranked against another system yet. The figures, with their scopes, sample sizes and ranges, are on inferenceproofs.ai and in the board file; this README quotes none, so it cannot drift from them.
Release v0.1 stays in releases/v0.1/ as a record (two systems on their native workloads); the site prints none of its figures (RFC-0018 SI-5).
Note
In v0.1, one control outcome sits outside its expected set and is kept as recorded: a C6 bit flip on FlashProve run 2026-11-08-…-b1-prover-h200-hosted-3 landed in a proof part the adapter's verify does not read, and was accepted. That is a gap in the control design, not a finding against FlashProve; the RFC-0008 amendment now restricts C5, C6 and C8 to the parts a verifier declares it reads. Stored records are never rewritten; a rerun is a new record.
The harness is one Python package, iproofs (Python 3.12 or 3.13), managed with uv. Everything below runs on a laptop, with no GPU.
git clone https://github.com/ai-verification-observatory/inference-proofs
cd inference-proofs
uv sync
uv run iproofs --versionCheck the whole repository the way CI does: schemas, claim cards, derivations, pins, licences, neutral language (RFC-0013).
uv run iproofs validate --all
make ci # lint, mypy --strict, tests, coverage gate, validateRebuild the Level 1 board from the stored run records and compare it byte for byte (docs/REPRODUCE.md):
uv run iproofs release board v1.0 --checkRecompute a stored run's derived metrics from its raw records and compare them bit for bit:
uv run iproofs derive --check results/v0.1/sash/2026-11-09-sash-sash-l31-8b-c1024-prover-h200-hosted-1Build the site from the newest frozen release (iproofs.site computes every value a page prints; web/build.py draws the pages; every page reads in full without script):
uv run iproofs site build # writes web/
uv run iproofs site build --out DIR # a complete site in any other directoryOptional extras install only where they are needed: --extra gpu (NVML sampling on a GPU host), --extra fetch (model download), --extra prompts (tokenizers). For a rerun on a GPU host, see docs/REPRODUCE.md, Tier 3 and docs/runbooks/gpu-run.md, Hosted execution.
Every command
| Command | What it does | RFC |
|---|---|---|
iproofs validate |
run every CI check on the given paths | 0013 |
iproofs run |
run jobs of one system on one workload | 0010 |
iproofs reverify |
verify a kept run again on a verifier host with no GPU | 0018 |
iproofs derive |
compute derived metrics of a run directory | 0004 |
iproofs controls |
run the negative-control suite against a run | 0008 |
iproofs machine-record |
print the machine record as JSON | 0006 |
iproofs fetch-model |
fetch a workload's weights and check every hash | 0014 |
iproofs prompts |
prompt sets | 0003 |
iproofs serving |
production-serving records | 0015 |
iproofs release |
benchmark releases | 0009 |
iproofs site |
build the static site | 0009 |
iproofs level |
level files, beacon rounds and rung prompts | 0018 |
iproofs fidelity |
the fidelity gate of a rung | 0018 |
iproofs results |
copy a hosted job's run into results/ |
0010 |
iproofs proof-build |
proof-build records | 0018 |
inference-proofs/
├── rfcs/ normative decisions (sources of truth, in order: RFCs, spec, PRD)
├── docs/
│ ├── PRD.md why, for whom, scope, roadmap
│ ├── SPEC.md index and executive summary of the specification
│ ├── spec/ sections 00 to 10, overview to glossary
│ └── runbooks/ GPU-host procedures
├── schema/ JSON Schemas, each with valid and invalid examples
├── harness/ the iproofs package and its tests
├── systems/ one adapter per system: Dockerfile, system.yaml, claim.yaml, controls
├── workloads/ pinned models, prompt corpus, serving records
├── reference/ hardware profiles and price basis
├── results/ run directories, one per run, never edited after the fact
├── releases/ frozen benchmark releases
└── web/ the public site, built from the newest release
| We do | We do not |
|---|---|
| put a claim card beside every number | publish a number without its scope, grade, tier and source |
| run provers at pinned commits, with documented settings only | patch or tune a prover (SASH's Level 1 pin is a pull request this project's owner wrote, not merged upstream: systems/sash/README.md) |
| report ranges and sample sizes | report a single "best" run |
| record failed runs with their exit code and last lines | drop or round away a result |
| apply the same controls to every verifier | let a system choose its own tests |
copy what systems state, and write not stated |
infer a security property |
| rank passed systems inside one rung by the order rule | rank across rungs, score, certify or crown |
Open an issue or a pull request (CONTRIBUTING.md). To add a system, follow the adapter guide, systems/README.md. A row on Level 1 also needs the Level 1 entries that guide describes in From an adapter to a row on Level 1.
Code is licensed under Apache-2.0. Specifications, data, results and reports are licensed under CC BY 4.0.
The boundary:
- Code (Apache-2.0):
harness/,systems/*/(exceptclaim.yaml,claims/andexternal.yaml),site/templates,schema/and CI configuration. - Documents and data (CC BY 4.0):
docs/,rfcs/,workloads/,reference/,results/,releases/,targets/,reports/,web/learn/(the text of the site's Learn pages), and claim cards and external figures. - A file that fits neither list is code.
