Docker risc0 v3 by toshihiko-okubo · Pull Request #6 · datachainlab/bonsai-local

toshihiko-okubo · 2025-10-07T03:29:08Z

Create a release workflow and Dockerfile for creating an image

Signed-off-by: Jun Kimura <junkxdev@gmail.com>

toshihiko-okubo · 2025-10-07T03:32:35Z

QA (CPU)

Docker Image

$ kubectl -n lcp get pod bonsai-local-dd7d58d96-57ptj  -o jsonpath='{.spec.containers[*].image}'
ghcr.io/datachainlab/bonsai-local:docker-risc0-v3 docker:28-dind%

Remote Attestation

# lcp enclave generate-key --enclave=/root/enclave.signed.so --target_qe=qe3
0x15d907a82c20f071d0b53ff60e1024353447fa52

# lcp attestation zkdcap --enclave=/root/enclave.signed.so --enclave_key=0x15d907a82c20f071d0b53ff60e1024353447fa52 --prove_mode=bonsai --bonsai_api_url=http://bonsai-local.lcp.svc.cluster.local:8080 --tcb_evaluation_data_number=20
[2025-10-07T04:55:40Z INFO  remote_attestation::zkdcap] run zkDCAP attestation with prover_mode=bonsai image_id=e5056aa7a8064abeb648b31d5efa8697a79d416b937cb917d1428cec91a56c67 enclave_key=0x15d907a82c20f071d0b53ff60e1024353447fa52
[2025-10-07T04:55:41Z INFO  remote_attestation::dcap] TCB evaluation data numbers: TCBInfo=20 QEIdentity=20
[2025-10-07T04:55:41Z INFO  remote_attestation::dcap] DCAP RA done: status=SWHardeningNeeded advisory_ids=["INTEL-SA-00615"] validity=(not_before: 1759811560, not_after: 1762400835) min_tcb_evaluation_data_number=20
[2025-10-07T04:55:42Z INFO  remote_attestation::zkdcap] running pre-execution
[2025-10-07T04:55:42Z INFO  risc0_zkvm::host::server::exec::executor] execution time: 232.950155ms
[2025-10-07T04:55:42Z INFO  remote_attestation::zkdcap] pre-execution done: exit_code=Halted(0) cycles=5729281
[2025-10-07T04:55:42Z INFO  remote_attestation::zkdcap] proving with prover mode: bonsai
[2025-10-07T05:29:24Z INFO  remote_attestation::zkdcap] proving done: elapsed=2021secs stats=SessionStats { segments: 7, total_cycles: 6422528, user_cycles: 5729006, paging_cycles: 0, reserved_cycles: 0 }
[2025-10-07T05:29:24Z INFO  remote_attestation::zkdcap] receipt verified

toshihiko-okubo · 2025-10-07T03:50:18Z

QA(GPU)

Host

OS

Ubuntu 24.04

Kernel

Linux 6.8.0-85-generic

GPU

$ nvidia-smi
Tue Oct  7 12:49:44 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.65.06              Driver Version: 580.65.06      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070        Off |   00000000:01:00.0  On |                  N/A |
|  0%   47C    P3             26W /  200W |    1000MiB /  12282MiB |     38%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Setup

Installing the NVIDIA Container Toolkit

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/1.14.1/install-guide.html#installing-the-nvidia-container-toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list \
  && \
    sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Prerequisites

https://github.com/datachainlab/cosmos-ethereum-ibc-lcp?tab=readme-ov-file#prerequisites

Build Docker Image

$ docker buildx build -t bonsai-local:cuda --cache-to=type=local,dest=/path/to/cache,mode=max --cache-from=type=local,src=/path/to/cache -f dockerfiles/cuda.Dockerfile --load .

Important

In my environment, I needed to set 19 for segment_limit_po2.
https://github.com/datachainlab/bonsai-local/blob/main/src/prover.rs#L121

Error(segment_limit_po2=20)

bonsai-local-cuda  | thread 'tokio-runtime-worker' panicked at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/risc0-zkp-3.0.2/src/hal/cuda.rs:246:14:
bonsai-local-cuda  | called `Result::unwrap()` on an `Err` value: allocation failed on nodes: 268435456 bytes
bonsai-local-cuda  |
bonsai-local-cuda  | Caused by:
bonsai-local-cuda  |     "out of memory"
bonsai-local-cuda  |
bonsai-local-cuda  | Stack backtrace:
bonsai-local-cuda  |    0: risc0_zkp::hal::cuda::RawBuffer::new
bonsai-local-cuda  |    1: <risc0_zkp::hal::cuda::CudaHal<CH> as risc0_zkp::hal::Hal>::alloc_digest
bonsai-local-cuda  |    2: risc0_zkp::prove::merkle::MerkleTreeProver<H>::new
bonsai-local-cuda  |    3: risc0_zkp::prove::poly_group::PolyGroup<H>::new
bonsai-local-cuda  |    4: risc0_zkp::prove::prover::Prover<H>::finalize
bonsai-local-cuda  |    5: <risc0_circuit_rv32im::prove::hal::SegmentProverImpl<H,C,F> as risc0_circuit_rv32im::prove::SegmentProver>::prove_core
bonsai-local-cuda  |    6: <risc0_zkvm::host::server::prove::prover_impl::ProverImpl as risc0_zkvm::host::server::prove::ProverServer>::prove_segment_core
bonsai-local-cuda  |    7: risc0_zkvm::host::server::prove::ProverServer::prove_segment
bonsai-local-cuda  |    8: <risc0_zkvm::host::server::prove::prover_impl::ProverImpl as risc0_zkvm::host::server::prove::ProverServer>::prove_session
bonsai-local-cuda  |    9: <risc0_zkvm::host::server::prove::prover_impl::ProverImpl as risc0_zkvm::host::server::prove::ProverServer>::prove_with_ctx
bonsai-local-cuda  |   10: <risc0_zkvm::host::client::prove::local::LocalProver as risc0_zkvm::host::client::prove::Prover>::prove_with_ctx
bonsai-local-cuda  |   11: bonsai_local::prover::Prover::prove
bonsai-local-cuda  |   12: bonsai_local::serve::{{closure}}::{{closure}}

run container

compose.yaml

services:
  bonsai-local-cuda:
    container_name: bonsai-local-cuda
    image: bonsai-local:cuda
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - RUST_BACKTRACE=full
    ports:
      - "8080:8080"
    command:
      - --listen-address=0.0.0.0:8080

docker compose up -d

Enviornment Variables

export SGX_MODE=SW
export BONSAI_API_KEY=xxx # dummy API Key
source /opt/sgxsdk/environment

exec RemoteAttestation(SIM)

LCP

$ gh repo clone datachainlab/lcp
$ cd lcp
$ git checkout v0.2.17 && make
$ ./bin/lcp attestation zkdcap-sim --enclave=./bin/enclave.signed.so --enclave_key=$(./bin/lcp enclave generate-key --enclave=./bin/enclave.signed.so --target_qe=qe3sim) --prove_mode=bonsai --bonsai_api_url=http://localhost:8080 --tcb_eval_data_num=1
[2025-10-07T05:17:43Z INFO  store::rocksdb] initialize a database: "/home/ubuntu/.lcp/state"
[2025-10-07T05:17:43Z INFO  keymanager] initialized Key Manager: "/home/ubuntu/.lcp/km.sqlite"
[2025-10-07T05:17:43Z INFO  remote_attestation::zkdcap] simulate zkDCAP attestation with prover_mode=bonsai image_id=e5056aa7a8064abeb648b31d5efa8697a79d416b937cb917d1428cec91a56c67 enclave_key=0xb1767d0d6c1ef9a86df512d924d63d4b88555746
[2025-10-07T05:17:43Z INFO  remote_attestation::dcap_simulation] DCAP RA simulation done: status=UpToDate advisory_ids=[] validity=(not_before: 1759814263, not_after: 3907297910) min_tcb_evaluation_data_number=1
[2025-10-07T05:17:43Z INFO  remote_attestation::zkdcap] running pre-execution
[2025-10-07T05:17:43Z INFO  risc0_zkvm::host::server::exec::executor] execution time: 103.1581ms
[2025-10-07T05:17:43Z INFO  remote_attestation::zkdcap] pre-execution done: exit_code=Halted(0) cycles=3417355
[2025-10-07T05:17:43Z INFO  remote_attestation::zkdcap] proving with prover mode: bonsai
[2025-10-07T05:17:59Z INFO  remote_attestation::zkdcap] proving done: elapsed=16secs stats=SessionStats { segments: 8, total_cycles: 4194304, user_cycles: 3417522, paging_cycles: 0, reserved_cycles: 0 }
[2025-10-07T05:17:59Z INFO  remote_attestation::zkdcap] receipt verified

bonsai-local

bonsai-local-cuda  | 2025-10-07T05:17:43.529522Z  INFO bonsai_local::routes: base_url: http://localhost:8080/
bonsai-local-cuda  | 2025-10-07T05:17:43.530371Z  INFO bonsai_local::routes: create_session: 5d2bc65f-7efc-4ae9-b162-7cc60ddbf00c
bonsai-local-cuda  | 2025-10-07T05:17:43.530422Z  INFO bonsai_local::prover: Received message: ProverMessage::RunSession: { task: Task { session_id: "5d2bc65f-7efc-4ae9-b162-7cc60ddbf00c", image_id: "e5056aa7a8064abeb648b31d5efa8697a79d416b937cb917d1428cec91a56c67", input_id: "caf170c9-eb26-46b3-8772-fd3af26c54ad", assumptions: [] } }
bonsai-local-cuda  | 2025-10-07T05:17:43.530429Z  INFO bonsai_local::prover: Running task...
bonsai-local-cuda  | 2025-10-07T05:17:43.716111Z  INFO risc0_zkvm::host::server::exec::executor: execution time: 111.980607ms
bonsai-local-cuda  | 2025-10-07T05:17:56.578550Z  INFO risc0_groth16::prove::cuda: shrink_wrap: 222668 seal bytes
bonsai-local-cuda  | Inputs loaded in 44ns
bonsai-local-cuda  | Graph loaded in 409.636655ms
bonsai-local-cuda  | Inputs populated in 38ns
bonsai-local-cuda  | generic typed graph calculated in 996.395028ms
bonsai-local-cuda  | 2025-10-07T05:17:59.441081Z  INFO bonsai_local::prover: Task done: "5d2bc65f-7efc-4ae9-b162-7cc60ddbf00c"
bonsai-local-cuda  | 2025-10-07T05:17:59.541263Z  INFO bonsai_local::routes: base_url: http://localhost:8080/
bonsai-local-cuda  | 2025-10-07T05:17:59.541652Z  INFO bonsai_local::routes: get_receipt: 5d2bc65f-7efc-4ae9-b162-7cc60ddbf00c
bonsai-local-cuda  | 2025-10-07T05:17:59.544683Z  INFO bonsai_local::routes: create_snark: 5d2bc65f-7efc-4ae9-b162-7cc60ddbf00c
bonsai-local-cuda  | 2025-10-07T05:17:59.545046Z  INFO bonsai_local::routes: base_url: http://localhost:8080/
bonsai-local-cuda  | 2025-10-07T05:17:59.545381Z  INFO bonsai_local::routes: get_receipt: 5d2bc65f-7efc-4ae9-b162-7cc60ddbf00c
bonsai-local-cuda  | 2025-10-07T05:18:33.767270Z  INFO bonsai_local: Cleaned up expired entries

bluele · 2025-10-07T06:53:45Z

+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-22.04-16core


Can't you use ubuntu 24 instead?

@toshihiko-okubo I just realized you are using the custom runner. Is it necessary to use a custom one for this job?

@toshihiko-okubo what do you think?

The size of the cuda Docker image exceeds the disk size of the GitHub-hosted runner, so we must use a self-hosted runner.

@toshihiko-okubo I understand, but you can use a more suitable runner from https://docs.github.com/en/actions/reference/runners/larger-runners .

you can use a more suitable runner

Is it correct to understand that we should use the minimum spec necessary for execution?

When using the current large runner (4-core CPU), a Docker build without cache may take approximately 35 to 40 minutes. Specifying a lower specification may cause concerns about increased delivery times.

Since the large runner incurs costs based on workflow execution time, the current configuration strikes a good balance between cost and performance.

@toshihiko-okubo I see, so you wish to increase not only the disk size but also the number of cpu cores. While I don't believe execution time is that critical for this task, it's fine for now.

bluele · 2025-10-07T06:57:42Z

+          feature='${{ matrix.feature }}'
+
+          suffix=""
+          # If the profile is not `release`, append -${profile}


Are L53 and L58 comments required? I think these are obvious.

bluele · 2025-11-01T02:12:29Z

+
+ARG FEATURES="cuda"
+ARG PROFILE="release"
+ARG CARGO_RISCZERO_VERSION="3.0.3"


I believe the default values for these versions are unnecessary. The builder should give explicit versions.

bluele · 2025-11-01T02:13:49Z

+
+WORKDIR /app
+
+ARG FEATURES="cuda"


The default feature should be either removed or "default".

bluele · 2025-11-14T02:20:30Z

+          - "release"
+        risczero_version:
+          - "3.0.3"
+        risczero_growth16_verifier:


s/growth16/groth16

Also, I prefer to use risczero_groth16_version same as risczero_version's format.

bluele

LGTM👍

bluele · 2025-11-15T15:15:44Z

@toshihiko-okubo before merging this, can you squash the commits into a single commit?

add Dockerfile

d7c91d3

Signed-off-by: Jun Kimura <junkxdev@gmail.com>

toshihiko-okubo self-assigned this Oct 7, 2025

toshihiko-okubo marked this pull request as ready for review October 7, 2025 05:55

toshihiko-okubo requested a review from bluele October 7, 2025 05:55

bluele reviewed Oct 7, 2025

View reviewed changes

Comment thread dockerfiles/cuda.Dockerfile

toshihiko-okubo force-pushed the docker-risc0-v3 branch 2 times, most recently from 4cd5163 to edd4ec5 Compare October 7, 2025 10:47

toshihiko-okubo force-pushed the docker-risc0-v3 branch 2 times, most recently from f949484 to 40f8ef8 Compare October 27, 2025 03:06

toshihiko-okubo requested a review from bluele October 27, 2025 11:18

bluele reviewed Nov 1, 2025

View reviewed changes

toshihiko-okubo force-pushed the docker-risc0-v3 branch 2 times, most recently from a8bcf15 to 5ad846c Compare November 13, 2025 05:47

bluele reviewed Nov 14, 2025

View reviewed changes

toshihiko-okubo force-pushed the docker-risc0-v3 branch from e97a15a to 9259789 Compare November 14, 2025 03:25

bluele self-requested a review November 14, 2025 07:50

bluele approved these changes Nov 14, 2025

View reviewed changes

add workflow to push docker image to ghcr.io

6a9752e

toshihiko-okubo force-pushed the docker-risc0-v3 branch from 9259789 to 6a9752e Compare November 17, 2025 03:13

bluele merged commit 788dc03 into main Nov 17, 2025
1 check passed


		WORKDIR /app

		ARG FEATURES="cuda"

Conversation

toshihiko-okubo commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toshihiko-okubo commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QA (CPU)

Docker Image

Remote Attestation

Uh oh!

toshihiko-okubo commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QA(GPU)

Host

OS

Kernel

GPU

Setup

Installing the NVIDIA Container Toolkit

Prerequisites

Build Docker Image

Error(segment_limit_po2=20)

run container

Enviornment Variables

exec RemoteAttestation(SIM)

LCP

bonsai-local

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bluele left a comment

Choose a reason for hiding this comment

Uh oh!

bluele commented Nov 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

toshihiko-okubo commented Oct 7, 2025 •

edited

Loading

toshihiko-okubo commented Oct 7, 2025 •

edited

Loading

toshihiko-okubo commented Oct 7, 2025 •

edited

Loading