Skip to content

Latest commit

 

History

History
153 lines (99 loc) · 35.7 KB

build.md

File metadata and controls

153 lines (99 loc) · 35.7 KB

What the prover needs, and how to build it

A path on this page is relative to prover/, the parent of the folder docs/; a command is run from prover/. In the text below, "this folder" means prover/. ../docs/glossary.md explains the terms, "full size" among them; a size that is marked "by formula" follows from the parameter file: it was computed by a script that is no part of the programs, not read off a file or a run. What this page says of build.sh, ../verifier/build.sh and check_nvcc_version.sh, and its table of environment variables, were compared with the sources once, in a text from which this page differs in places, and that comparison found sentences of that text false, none of which stands on this page in that form; the rest of the page was not compared, nor were the sentences that tell of runs or measurements or of what NVIDIA states.

What you need

  • Linux 4.14 or later on x86-64 with the SHA instruction extensions, and bash 4.4 or later; in a container or sandbox, one that lets the call madvise(MADV_WIPEONFORK) through. Elsewhere cp_commit and cp_prove stop with a sentence that names the call and the version ("the operating system gave no memory page that a fork wipes: mmap() or madvise(MADV_WIPEONFORK) failed; the latter needs Linux 4.14 or later"). (build.sh compiles cp_commit's SHA-256 code, make_model's and some unit tests' for these instructions, and none of them looks whether the CPU has them: without them such a program ends with the shell's bare "Illegal instruction". cp_prove's SHA-256 code is compiled as plain C++; cp_verify and weight_sums/ do not use the instructions.)
  • CUDA 13.2 Update 2, that is nvcc 13.2.86: the release with which the programs are built and tried. A later toolkit is not supported; check_nvcc_version.sh lets a later release through, with one line of note on standard error. nvcc is looked up on PATH first and then in /usr/local/cuda/bin. Every script that compiles GPU code first runs check_nvcc_version.sh, which stops with one line if the compiler is older. The CUDA Toolkit is not part of this repository; NVIDIA distributes it under its End User License Agreement. (Lines that install it; why this release, and what a build checks of the compiler.)
  • gcc and g++ 13, as the first programs of those names on PATH (nvcc compiles host code with gcc and links with g++)
  • one H100 80 GB (sm_90a), and an NVIDIA driver that is installed and works: nvidia-smi must list the H100 (the driver's library). By NVIDIA's page, the lowest driver for CUDA 13 is 580, though not everything then works (more); the programs have run under driver 580†, and the lowest driver that works is not known.
  • python3 and a build of ../verifier. Its program cp_verify checks a proof; cp_commit runs it on the files that it makes, and whoever checks a response before handing it on needs one built from the same sources as cp_prove.

One way to get that compiler. That compiler can be had from NVIDIA's own Python packages on the public package index (PyPI), without NVIDIA's installer. The lines below put these packages into a directory of your own (about 1 GB, and about half a gigabyte more in pip's cache under the home directory) and add what the link step needs and the packages lack: the plain names of three libraries, and the name libcuda.so, as a link to this machine's own driver library (so the NVIDIA driver must be installed). The third line needs pip (on Ubuntu: the package python3-pip).

  • Type the lines one by one and stop at the first that fails: nothing joins them, and each needs what the lines before it made. Eight of them stop with a sentence if cuda or lib is not set: without that guard pip, given an empty target, installs into the current Python, as root into the system's (read in one version of pip's code). The guard does not help where cuda or lib is still set from other work and the line that sets it is left out: the lines then act on that directory.
  • The second line makes that directory, so that it is new: if one of that name exists already, mkdir says so, the line fails, cuda and lib are unset, and every later line that needs one of them stops with its sentence ("first run the line with ..", which is then not the cause) and touches nothing. If the second line fails, do not go on and do not leave it out: give another name in the first line and start again from there. With the first line typed again and the second left out, nothing stops the later lines: they act on the directory that was there, and chmod takes write permission from group and others in all of it (seen). The chmod line is there because, where your umask leaves the group write permission, build.sh would otherwise give the programs no run path.
  • The packages are NVIDIA's, under NVIDIA's licence, which is yours to read and accept.
  • In zsh, type the last line without the words from # on. In every new shell the first line and the line with export are needed again.
cuda=$HOME/cuda-13.2.86
mkdir "${cuda:?first run the line with cuda=}" || { unset cuda lib; false; }
python3 -m pip install --target "${cuda:?first run the line with cuda=}" "cuda-toolkit[nvcc,cublas,cccl,crt,nvvm,cudart]==13.2.2"
lib="${cuda:?first run the line with cuda=}/nvidia/cu13/lib"
for n in cublasLt cublas cudart; do ln -s lib$n.so.13 "${lib:?first run the line with lib=}/lib$n.so"; done
mkdir "${lib:?first run the line with lib=}/stubs"
ln -s "$(/sbin/ldconfig -p | awk '/libcuda\.so\.1 .*x86-64/ { print $NF; exit }')" "${lib:?first run the line with lib=}/stubs/libcuda.so"
chmod -R go-w "${cuda:?first run the line with cuda=}"
export PATH="${cuda:?first run the line with cuda=}/nvidia/cu13/bin:$PATH"
nvcc --version    # the last line but one ends: V13.2.86

Next: Build. With a toolkit installed this way, the prover's build prints three notes that the CUDA libraries will be loaded from a directory that belongs to your account and not to root: they are expected.

The details stand behind Build, under "What you need: the details":

Build

Run this from prover/. build.sh takes one target per call and writes into build/, or into the folder named by the environment variable OUT. A relative OUT is taken from prover/, wherever you call the script from (verifier/build.sh takes its relative output directory from the directory in which you stand): give OUT as a full path, outside prover/ or below prover/build, since files elsewhere inside prover/ can count as sources in the digest of every later build record.

../verifier/build.sh
./build.sh            # THE programs: cp_commit, cp_prove, cp_verify in build/hc; they alone take a model's files
./build.sh tests      # for tests: cp_prove and cp_commit WITH the test options in build/hc/test_build, and make_model

The verifier comes first, because the target cp_commit puts a copy of ../bin/cp_verify next to the program, which runs it. Without ../bin/cp_verify the target cp_commit fails after its compile, with status 1. The target cp_prove copies no cp_verify. The default build asks at its head, before the half hour of compiling (status 2), and once more of the copy (status 1), that ../bin/cp_verify was built from these very sources, and refuses another; a test build takes whatever is in ../bin, so there an older verifier can stand beside a newer cp_commit: build the verifier again first, every time.

The default build refuses a second call into a folder where one is still running. It cannot tell a lock that is held from a lock that cannot be taken: where flock cannot lock a directory, as on some network file systems, it says that another call of build.sh is building and ends with status 2 although none is (read in the script): set OUT to a directory on a local disk. A caller can get past that lock by the variable BUILD_SH_HOLDS_THE_LOCK, and there is no lock where flock is missing.

target file what it does
cp_prove hc/test_build/cp_prove P: proves one batch of requests, or serves batch after batch, against a weight commitment
cp_commit hc/test_build/cp_commit C: commits to a model's weights
make_model make_model a CPU program that writes a small synthetic model as GGUF files (GGUF is a public file format for model weights)

The table's targets make the programs WITH the test options. The service needs the default build's cp_prove and cp_commit, and cp_commit the copy of cp_verify beside it. The word all is refused, with a line that names the two builds. The comment at the top of build.sh tells these targets and the default build's; every other target builds a test program or a benchmark and is described at its branch of the script's first case statement, and the usage line has all their names. Each program has a usage text, and that text is the reference for its options: all three print it at a command line that they do not take, cp_prove also with -h. The default build's cp_prove prints ordinary sentences too (the usage text; each refusal with its reason; the lines of the set-up) until it is about to read its first secret; from there on only fixed lines such as p258 end 1 reach its output (closed_front.md).

What a build writes, installs and removes. A build writes nowhere but below OUT, with one exception: only the default build's targets start the compilers with a reduced environment: of yours they see PATH alone, and their TMPDIR is the build's own work directory. In every other target the compilers see yours: with DEPENDENCIES_OUTPUT or SUNPRO_DEPENDENCIES set, g++ appends to the file that the variable names, outside OUT (read in the script, not run); unset both before such a build.

OUT must be a directory, no symbolic link, that belongs to you (or to root, or under sudo to the account that ran sudo) and that neither its group nor other users can write to, because what is built there is run; if it does not exist, build.sh makes it, with what is missing above it, and whatever it makes is closed to group and others for writing. Nor may OUT hold anything that belongs to another account, or a directory that its group or other users can write in. That look asks who owns each entry and, of directories only, who can write in them: it does not ask the mode of a regular file of your own, and it does not follow a symbolic link of your own that leads out of OUT. Nor may a place above OUT be one that another account may be able to change: a directory that its group or other users can write in without the sticky bit, or one that belongs to another account than you, root or, under sudo, the account that ran sudo. ACLs are read in none of these looks, at OUT, at what it holds or at the directories above it: only owners and mode bits count. All that holds for an OUT that you name, and for the default, inside the checkout, too, unless a place above it is open in that way: there build.sh prints a note instead and builds, since whoever can change such a place can change the script and its sources too. (A clone made under a umask of 002 has group-writable directories: the script's message names the directory and says how to close it.) Each call of build.sh makes a new work directory there, building.XXXXXX; the compilers and the script write their files only into it, temporary files too. After every step and check has passed, renames give the target's files their names, and the work directory goes. So a target that fails or is interrupted installs nothing and removes nothing of an earlier build (that holds of a single call; of the default build's two calls, and Ctrl-C, see "Interrupting a build"). ./build.sh with no target word, and ./build.sh tests, call the script once for each program, and each call installs its own: if one of them fails, OUT can hold programs of two builds, and the last line says so.

When the default build's cp_commit is installed, an older copy of cp_verify beside it goes, and the new copy comes at the end of ./build.sh: in between, a cp_commit that runs from that directory finds no cp_verify. ./build.sh serving_cp_commit, called by itself, also removes the cp_verify in OUT/hc and installs none; it ends with status 0 and prints no line about it. Run the default build afterwards (./build.sh, or ./build.sh serving): only it installs the new copy. ./build.sh serving_cp_prove, called by itself, leaves the cp_verify in OUT/hc as it is, so a new cp_prove can stand beside an older cp_verify from other sources, and no record ties the two: cp_prove's build record has no line on a verifier, though the digest in it covers verifier/src, from which cp_prove compiles nothing.

After a build under sudo, the directories that it created below OUT are root's: a later build without sudo into the same OUT can pass every check, compile for the whole half hour and fail only when it installs (read in the script). Build with sudo every time or never, or set OUT to another directory. A work directory can stay behind, as after SIGKILL, and can be removed once no build runs; a default build that was interrupted with Ctrl-C has left its own in OUT (seen once; the cause is not known). Two calls at once into one OUT take nothing of each other's away; for the default build a lock keeps them apart where the tool flock is installed.

Warnings. The compilers print warnings on the way, and a warning does not stop a build. With nvcc 13.2.86 the default build prints warnings of three kinds, from source files of this repository, none from the toolkit's headers (reported†). Two kinds are expected. #20011-D (common/zeroing_heap.h): the programs replace the global operator new and operator delete by routines that zero memory before it is freed; nvcc takes these operators for functions of the host and the GPU and objects that they call a function of the host. #550-D (field/lazy_gl64.cuh, field/transform_groups_gl64.cuh): scratch registers of four blocks of inline PTX, which are written and read only inside the block. The third kind, #177-D (prover/layer_prove.cuh, prover/layer_walk.cuh, hiding/forest_fused.cuh, attn/front.cuh), names seven local variables that nothing reads. g++ prints no warning for cp_verify. A warning of another kind, or from another file, is worth a look.

A good end of ./build.sh with no target word, or with serving, has status 0 and the last line THE PROGRAMS ARE BUILT: ..; ./build.sh tests ends with the line on the run path, or the warning (below); every call that has compiled with nvcc prints Compiled by <nvcc's path>, release <X.Y>, V<X.Y.Z> near its end, and for the other targets that is the last line, or, where the compiler's sample has run on a GPU, the last but one; ./build.sh with no target word shows the line three times, once for each of its two inner calls and once for itself. The call took some 26 to 35 minutes† on 11 or more cores in the few runs that were timed, none of them on an idle machine; one of the script's first lines on the screen says "about half an hour in all". It can be silent for a long time: in two runs no line came to the screen for 16 and for some 20 minutes (reported†).

One build of the format. A digest has 8 words of 32 bits. That is a constant of the build (the macro DIGEST_WORDS; hash_commitment/hf_table_format.h), never a line of a file and never a value of a run, and nothing else can be built: the sources hold static_assert(DIGEST_WORDS == 8) (hash_commitment/hf_table_format.h; ../verifier/src/vf_digest_words.h), ./build.sh takes --dw 8 alone, as a word that changes nothing, and ../verifier/build.sh has no such option.

There is one field, p = 2^64 - 2^32 + 1 (number 2): --field 2 before the target, in both scripts, may be said and changes nothing; any other number is refused.

What cp_commit does is told in ../docs/architecture.md, "From weights to a checked proof"; its options are in its usage text; what to do when it is killed or fails: ../docs/faults.md, "A killed or failed cp_commit".

./build.sh serving is the same as ./build.sh with no target word: the default build. ./build.sh -h or --help prints one line with the targets' names and a second that points to the script's head and to this page, on standard output, and ends with status 0. A first argument that is no target gets that line on standard error, and status 2 (all keeps its own sentence). In both cases nothing is created, checked or compiled. The main targets are told in the script's first comment, the others at their branches of its first case statement.

The GPU kernels are CUDA code and use NVIDIA's public libraries (cuBLAS, cublasLt, CUB, wmma, the pipeline primitives). Where a file's own comment says that its code follows or is like public code, ../THIRD_PARTY.md quotes that comment under the work's name, and ../docs/sources-notes.tsv has most of the same matter file by file. The code that decodes the block formats Q4_K, Q5_K and Q6_K byte by byte is adapted from ggml's public code (MIT licence; ../THIRD_PARTY.md, "ggml", where one inline PTX statement of weights_blocks/row_pass_fused.cuh is told too). The storage types Q8_0, F16 and BF16, which the programs read too, are also ggml's. Source files cite published methods in their comments, and ../PUBLISHED-WORKS.md gives some of these citations in full ("NVIDIA's manuals, library documentation and blog posts"; "Published methods of the provers and the scripts"). Other places where code follows public code: ../THIRD_PARTY.md, "BLAKE2b", "ChaCha20", "MegEngine", "NVIDIA CUTLASS".

Interrupting a build

Ctrl-C and ./build.sh with no target word: the script runs two calls of itself side by side, the one that builds cp_prove in the background, and a script's shell starts a background command with the interrupt ignored. Where env takes the option --default-signal (GNU coreutils 8.31 or later; the script asks first), it starts that call with the interrupt set back to its default, so Ctrl-C at a terminal ends the whole default build: each call that still runs says that it failed with status 130 and installs nothing, unless it is already installing, and the script ends with status 130. In six trials† on real default builds, 5 to 900 seconds in, on a machine without a GPU, the script ended within a second, and two seconds later no process of the build was left. A call that had already ended well has installed its program (cp_commit is done first, after some minutes, cp_prove later), and the closing line says so: OUT may then hold programs of two builds, so build again. If the script itself was started with the interrupt ignored (behind & in another script, or by a runner that ignores it), no call of the build hears the interrupt: send SIGTERM to each call.

Where env does not take the option, the background call itself does not receive the interrupt; the program that it is running at that moment may. If the compiler dies of the signal, the whole build ends within a second with status 130, and nothing is installed (seen once†). If the interrupt comes while that call runs a program that does not die of it, the script prints build.sh: waiting for a part of the build that still runs in the background and waits, for up to the rest of the half hour, without hearing a further interrupt; if that call then ends well it installs the new cp_prove. To stop it then, send SIGTERM to that one process, whose command line names serving_cp_prove: it installs nothing, unless it is already installing; compilers that it has started may run on for some minutes. On both kinds of machine a signal that is sent to the script's process alone (kill; a supervisor's TERM) does not reach its calls, which may go on for up to half an hour and install their programs (read in the script; no machine with an older env was tried, nor an interrupt in the moment of an installing).

What you need: the details

The compiler. CUDA 13.2 Update 2, that is nvcc 13.2.86: the release with which the programs are built and tried. All builds with that toolkit used it as installed from NVIDIA's Python packages for the compiler, the runtime and the C++ core libraries; these hold no NVTX headers. A build on the whole toolkit as NVIDIA's installer installs it, which most likely holds them, has not been tried.

CUDA 13.4 (nvcc 13.4.59) is not supported. The toolkit's NVTX headers, which a header of the CUDA C++ Core Libraries (CCCL) takes in wherever they are installed, bring getenv() and dlopen() into a program: NVTX's start-up code reads the variable NVTX_INJECTION64_PATH and loads the library that it names (reported). So build.sh passes -DCCCL_DISABLE_NVTX on every nvcc command, under which CCCL's header does not take NVTX's headers in, and it does not install a default-build program one of whose printable strings is that variable's name. That reaches the programs' own code, not the libraries that they load. Under that toolkit only verifier/build.sh, the default build and one small example on made-up weights have been built and run, on one machine image†; the test build, the tests, every other configuration and the full size have not.

Why that release: the first reason is a fault of the compiler that NVIDIA itself reports: its release notes for CUDA 13.2 Update 2 list it among the faults repaired in that release ("Resolved Issues", "CUDA Compiler") and date it back to CUDA 12.8. By that note, where the threads of a warp branch apart a second time inside a branch, the compiled code can lose what a register should hold, and the kernel's answer can then be wrong.

The second reason: nvcc 13.0.88 miscompiles one small kernel, without any message, and nvcc 13.2.86 compiles it correctly; no release note that has been found mentions that bug, so it is not known whether the later release repairs it or only compiles this kernel differently. build.sh checks the compiler in use on that kernel, by value: before every target that runs nvcc it compiles compiler_sample.cu, whose head says what it checks, with its own compile line and, if the machine has a GPU, runs it and compares the results with the CPU's. A wrong value stops the build. So does a sample that cannot be started or is killed. If all values are right, one line on standard output says so: it begins build.sh: the compiler passed the check by value and repeats the sample's own line.

Without a GPU, or if the GPU cannot be used (for example, it has no free memory), one line on standard error says that the sample was built but not run to its end, and the build goes on with the version check alone. Most of the runs that these pages tell used programs from builds made that way, on machines without a GPU: in those builds the sample compared nothing, and only the compiler's version was checked (reported, from the builds' own logs). That line on standard error is also what the build rows of the test table print, on any machine, unless the runner is given --unconfined: run_suite.py gives a row that builds no access to the GPU; its row gpu.compiler_sample runs the same program there. So a build on a machine with a GPU uses that GPU for a moment.

To keep a build off the GPU, set CUDA_VISIBLE_DEVICES to the empty string for the call (CUDA_VISIBLE_DEVICES= ./build.sh): the sample then finds no GPU, starts nothing there, and the one line is printed. The run has no time limit: on a GPU that no longer answers, build.sh waits without a word. A right result shows nothing about other kernels. A compiler that passes both checks may have other bugs; NVIDIA's notes list open ones under "Known Issues".

What has been tried with a toolkit installed this way (reported), each walk† with pip already on the machine and the packages taken from a mirror, not from the public index itself. The ten lines as printed, on a machine with an H100 and the driver and on one with neither: every line ended with status 0, but for the link to the driver's library, which the second machine lacks. ../examples/quickstart-gpu.sh with such a toolkit, on one H100: exit code 0. The quick start up to its full-size example, on synthetic weights, three times, each time with one H100: it ended as ../docs/quickstart.md shows; a CUDA 13.4 toolkit stood on those machines too, on PATH before the walk, and which CUDA libraries the programs loaded was not looked at. Not tried: a service with several requests in a batch or over a long time, the GPU tests, a real model's weights. In a minimal Ubuntu 24.04 system (made with debootstrap) python3 -m pip fails with "No module named pip": python3-pip is not installed there.

At full size. The full-size example configuration with the exact match is examples/target_full_f2; examples/TARGET-KITS.md has the sizes of the key file, a leaf file and a proof under its policy.txt, by cp_verify's planning call. The kept nodes of a weight table (hcnodes.bin, one file for the anchor and one for each leaf table) have 1,073,741,824 bytes each (1.1 GB, by formula: 32 bytes for each of the 2^25 nodes of level 7, the default level). Measured times and memory at full size: ../guide/performance.md.

Where the programs find the CUDA libraries. cp_prove and cp_commit, in both builds, link the shared CUDA runtime (libcudart.so) and cuBLAS, and build.sh passes -lcublasLt to every link that nvcc makes. build.sh gives every program that nvcc links a run path to the library directory of the toolkit that compiled it (lib64 next to nvcc's own directory, where NVIDIA's installers put the libraries, or else lib, where its Python packages put them; with links resolved), so the programs start without LD_LIBRARY_PATH and without an entry in the loader's configuration. LD_LIBRARY_PATH would not serve: the default build's cp_prove, and both programs in every build in every run that is neither on made-up weights under a test's seed nor refused before that step (a fault of the command line, a test's variable in the environment, a build for tests on a model's files), start themselves a second time with a reduced environment, in which LD_LIBRARY_PATH is not kept (common/reduced_env_start.h). readelf -d <program> | grep -E 'RPATH|RUNPATH' shows the directory. Where it says RUNPATH, as most linkers write it, an LD_LIBRARY_PATH that is set is searched before that directory; the second start has none.

build.sh sets the run path only if that directory, every directory above it and every lib*.so* in it belong to root or to the account that builds, and none has a write bit for its group or for others (ACLs are not read), and every link among these libraries leads to a file in that same directory. Otherwise it prints one line that begins build.sh: warning: the programs get no rpath to the CUDA libraries:, with the reason, and builds on; ./build.sh with no target word and ./build.sh tests print it once more at their end, or else, on standard output, cp_prove and cp_commit carry a run path to <directory>: they load the CUDA libraries from there. Programs without a run path start only where the loader finds the libraries through its own configuration (ldconfig -p | grep -E 'libcudart|libcublas' shows what it knows; where it lists them, nothing more need be done): otherwise build with a toolkit in a directory that meets these conditions. build.sh looks once, when it builds: an owner or a mode that changes afterwards is not seen, and the programs make no such check when they start.

Whoever can write to the directory of the run path chooses code that the programs load. If the toolkit is root's, that is root alone. If it belongs to the account that builds, nothing is lost as long as the same account runs the programs where they were built, since that account can replace the programs themselves. It matters in two cases: another account (root, a service account) runs the programs; or the programs are copied into a place that the building account cannot write, or to another machine, while the run path still names the old directory. In both, the building account, or on another machine whoever can create that path, can put code into a process that holds the seed and the weights. For each program of the default build, build.sh prints a line that begins build.sh: note: and names the first directory or library that is not root's, if there is one; ./build.sh prints it once more at its end.

The driver's library, libcuda.so.1, is not in the toolkit; the loader finds it through its own configuration wherever the NVIDIA driver is installed in the usual way. Where a container finds it only through LD_LIBRARY_PATH, the second start does not find it, and the program fails there; the driver's directory must then stand in the loader's configuration of that container. The cp_prove of the build with the test options (./build.sh tests) makes no second start on a commitment to made-up weights under a test's seed, so such a run does not show whether the libraries are found in the reduced environment; every other run that is not refused first, and every proving or serving run of the default build (./build.sh with no target word), starts a second time.

Which CUDA libraries the programs have run with

A build by the steps above, where build.sh sets the run path, loads the runtime and the cuBLAS of the toolkit that compiled it: with CUDA 13.2 Update 2 that is cuBLAS 13.4.1.3. Runs† were mostly not of that pairing: their programs were compiled by nvcc 13.2.86, one set against the headers of cuBLAS 13.1.0.3, which belongs to CUDA 13.0, another against those of 13.4.1.3, and nearly all of them loaded cuBLAS 13.1.0.3, under driver 580.

By NVIDIA's page on minor version compatibility (docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html), two things beside a program need not come from the toolkit that compiled it, provided that the first number of the CUDA release is the same. One is the driver: any will do from the lowest that NVIDIA names for that first number (580 for CUDA 13), though not everything then works. The other is the CUDA libraries that the program loads: they may be of an earlier or a later release than the compiler's, but only such as NVIDIA makes this promise for. The page has no list of these.

By the make of the calls the library's release should not move a byte of a commitment or a proof: every call of cuBLAS and cuBLASLt in the programs but three multiplies 8-bit integers into 32-bit sums, which is exact as long as no sum overflows (the method, exact products of integer matrices through a numerical library with a bound on the sums, is public: J.-G. Dumas, P. Giorgi, C. Pernet, "Dense Linear Algebra over Word-Size Prime Fields: the FFLAS and FFPACK packages", arXiv:cs/0601133; ../PUBLISHED-WORKS.md): at every place that was followed a test in the code bounds the length, in the calling routine or in cp_commit's and cp_prove's tests on the parameter file.

Tried†, on made-up weights in test builds under a public seed: at the quick start's small settings, the same programs on cuBLAS 13.1.0.3 and on 13.4.1.3 wrote the same bytes, once on each side; at full size, once, by report, they wrote the same bytes on both (the key and leaf files, the secret folder, 25 proofs and replies). Which libraries were taken was read from the loader's listing before the runs, not from a running program. Under a test's seed cp_commit leaves its worst-case check out unless the call asks for it, and by the same report it was left out at full size, so that run says nothing of it.

Not tried on 13.4.1.3: the script's check of the libraries on a machine that has this toolkit alone (that machine's loader's list also held older CUDA libraries), the GPU tests, a toolkit installed by NVIDIA's installer, a service at full size.

The three other calls are in cp_commit's worst-case check, which multiplies in binary64 (weights_commitment/worst_case.cuh): such sums depend on the order in which a release adds them. In the trial at small settings above, in which the check was made, its table came out the same under both releases, at three decimals and one small shape. At a site near its limit two releases could still part: cp_commit would then end with status 7 on one machine and commit on another, and ../scales/worst_case.py, which reads from that table how many whole bits each site is short, could give two files of scales, and so, once a parameter file is made from them, two commitment IDs (reasoned, not seen). By the same reasoning that would be a matter of use, not of soundness or hiding. The programs ask no library for its version.

Environment variables that the scripts read

What a caller may set for the build scripts, the example, the test runner and the audit tool; each script's own head has the details. The table is not every variable that a script of this repository reads: the two scripts under ../examples/ read CUDA_VISIBLE_DEVICES too, ../examples/quickstart-gpu.sh and tests/unit.sh read TMPDIR, and test scripts read more (GPU_RUN in several of them; MAKE_MODEL, SMALL_MODEL and others), each told in the script's head. Paths are relative to prover/, as on the rest of this page.

variable read by what it does if it is not set
OUT build.sh the directory into which a build writes ("What a build writes, installs and removes", above, has the rules for it) build/
CUDA_VISIBLE_DEVICES build.sh set to the empty string, it keeps a build off the GPU: the compiler's sample is then built and not run ("What you need", above) the sample runs on a GPU for a moment, if the machine has one
EIDB, ETAU build.sh, for targets with the test options only two constants of the format for test builds; the default build refuses to start if either is set the format's own values
BUILD examples/run_target_example.sh the folder with cp_commit, cp_prove and cp_verify build/hc, where build.sh with no target word puts them
BUILD ../run_suite.py the folder into which the GPU programs' test builds go, so that it outlives the run: a full path of letters, digits and / . _ - a folder below the run's work folder
GPU_RUN examples/run_target_example.sh a command prefix that is put before the two programs that use the GPU no prefix
GPU_LOCK ../run_suite.py the lock file for the GPU, one for all runs that share it; the runner makes it gpu.lock in the work folder, so two runs in two work folders do not wait for each other
GPU_PREFIX ../run_suite.py words that follow the lock's before every command that uses the GPU; it must not name the lock none
AUDIT_LIB ../tools/audit/audit.py, called by hand the path of the audit's small library, which is loaded into the process that holds the secrets: name your own build only libfield.so beside the tool; the two audit scripts build it and set the variable themselves

Under sudo, build.sh, cp_prove and the other scripts that look at the directories above a folder that they are given also read SUDO_UID, which sudo sets: a directory may then belong to the account that ran sudo. What the scripts under ../integrations/ read (some of vLLM's own variables among them) is told on that folder's pages.

† Not on the released programs. What it ran on, and how far that speaks for them: what was run on these programs, and what was not.