Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 2 additions & 27 deletions .cursor/rules/project-structure.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,9 @@ alwaysApply: false
- **benchmark/**: Go-based benchmark runner and core logic
- **contracts/**: Solidity smart contracts for testing
- **configs/**: YAML configuration files for different benchmark scenarios
- **report/**: React-based web UI for viewing benchmark results
- **runner/**: Go-based execution engine
- **scripts/**: Utility scripts for setup and automation

### Report UI Structure ([report/](mdc:report))
```
report/
├── src/
│ ├── components/ # Reusable UI components
│ ├── pages/ # Page-level components
│ ├── hooks/ # Custom React hooks
│ ├── utils/ # Utility functions and formatters
│ └── types.ts # TypeScript type definitions
├── public/ # Static assets
└── package.json # Frontend dependencies
```

## Configuration Management

### Benchmark Configs ([configs/](mdc:configs))
Expand All @@ -44,12 +30,6 @@ report/

## Development Workflow

### Frontend Development
1. Work in the `report/` directory for UI changes
2. Use TypeScript for type safety
3. Follow React functional component patterns
4. Use Tailwind CSS for styling

### Backend Development
1. Work in `benchmark/` and `runner/` directories
2. Use Go for performance-critical code
Expand All @@ -64,12 +44,6 @@ report/

## Build and Deployment

### Frontend Build
- Vite-based build system
- Static asset generation
- Optimized bundle output
- Development server with hot reload

### Backend Build
- Go module-based dependencies
- Cross-platform compilation
Expand All @@ -82,5 +56,6 @@ report/
1. Configuration files define test parameters
2. Go runner executes benchmarks
3. Results stored in structured format
4. React UI displays and analyzes results
(see [docs/report-data-contract.md](mdc:docs/report-data-contract.md))
4. The benchmark UI in `base/ui` reads that data and displays it

Binary file removed .github/assets/run-switcher.png
Binary file not shown.
15 changes: 0 additions & 15 deletions .github/workflows/examples.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,24 +86,9 @@ jobs:

./.github/scripts/run-example-benchmarks.sh ${{ runner.temp }}

- name: Build Report
run: |
cp -r ${{ runner.temp }}/output/ ./output/
pushd report
npm install
npm run build
popd

- name: Upload Output
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
if: always()
with:
name: output
path: ${{ runner.temp }}/output/

# upload to github actions artifact
- name: Upload Report
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: report
path: report/dist/
41 changes: 0 additions & 41 deletions .github/workflows/nodejs.yaml

This file was deleted.

40 changes: 8 additions & 32 deletions .github/workflows/public-benchmarks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -81,36 +81,12 @@ jobs:
--reth-bin ${{ runner.temp }}/bin/reth \
--geth-bin ${{ runner.temp }}/bin/geth

- name: Build Report
run: |
cp -r ${{ runner.temp }}/output/ ./output/
pushd report
npm install
npm run build
popd

- name: Upload static files as artifact
id: deployment
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3.0.1
# The report UI that used to be built here and published to GitHub Pages
# was removed from this repo; the run output is uploaded as a workflow
# artifact instead. See docs/report-data-contract.md for the schema.
- name: Upload Output
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
if: always()
with:
path: report/dist/

deploy:
runs-on: ubuntu-latest
needs: basic-benchmarks
permissions:
pages: write
id-token: write

environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@002fdce3c6a235733a90a27c80493a3241e56863 # v2.12.1
with:
egress-policy: audit

- name: Publish Report
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4.0.5
name: output
path: ${{ runner.temp }}/output/
8 changes: 0 additions & 8 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,3 @@ build-base-reth-node:

.PHONY: build-binaries
build-binaries: build-reth build-geth build-base-reth-node

.PHONY: build-frontend
build-frontend:
cd report && yarn build

.PHONY: run-frontend
run-frontend:
cd report && set -a && [ -f ../.env ] && . ../.env && set +a && yarn dev
37 changes: 14 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,21 +27,20 @@ Base Benchmark is a performance testing framework for Ethereum execution clients

## Results

Public results are available at the following links:
Benchmark runs write JSON to an output directory (see
[docs/report-data-contract.md](docs/report-data-contract.md) for the schema).
The `run-publish-benchmarks` workflow runs the public benchmark suite on every
push to `main` and uploads its output as a workflow artifact.

| Network | Link |
| ------------ | ---------------------------------------------------------------------- |
| Devnet | [https://base.github.io/benchmark/](https://base.github.io/benchmark/) |
| Base Sepolia | Coming soon |
| Base Mainnet | Coming soon |
This repo no longer ships a dashboard for viewing that output; it is a
benchmark runner and a data contract.

## Features

- **Performance Evaluation:** Test both block building and validation performance across execution clients (Geth, Reth, and more)
- **Comparative Analysis:** Measure client behavior across various inputs and workloads
- **Metric Collection:** Track critical metrics including submission times, latency, and throughput
- **Flexible Workloads:** Configure transaction patterns to match your specific needs
- **Interactive Dashboard:** Generate beautiful HTML reports with charts and run comparisons
- **Import & Merge:** Combine benchmark results from multiple machines with flexible tagging

## Repository Structure
Expand All @@ -65,16 +64,13 @@ Public results are available at the following links:
│ └── public/ # Production-ready benchmarks
├── contracts/ # Smart contracts for testing
│ └── src/ # Solidity source files
├── report/ # Interactive dashboard
│ └── src/ # React TypeScript application
└── clients/ # Client build scripts
```

## Prerequisites

- **Go:** Version 1.21 or later. Install from [go.dev](https://go.dev/dl/)
- **Foundry:** For smart contract compilation. See [installation guide](https://book.getfoundry.sh/getting-started/installation)
- **Node.js:** Version 18+ for the interactive dashboard. Install from [nodejs.org](https://nodejs.org/)

## Getting Started

Expand Down Expand Up @@ -119,15 +115,12 @@ To see available options:
./bin/base-bench run --help
```

### 5. View Results in the Interactive Dashboard
### 5. Read the Results

```bash
cd report/
npm install
npm run dev
```

Open your browser to the URL shown (typically `http://localhost:5173`).
The run writes one directory per benchmark under `--output-dir`, each holding a
`metadata.json` and per-role `metrics-<role>.json` timeseries.
[docs/report-data-contract.md](docs/report-data-contract.md) documents the
layout and every field, so you can consume it from your own tooling.

## Available Benchmarks

Expand Down Expand Up @@ -241,11 +234,9 @@ OPTIONS:

### Understanding Runs and Suites

When you view benchmark results in the interactive dashboard, you can switch between different test runs using the run switcher:

<div align="center">
<img src=".github/assets/run-switcher.png" alt="Run Switcher" width="600">
</div>
A suite is one execution of `base-bench run`. Each suite writes its own set of
runs to the output directory, keyed by the `BenchmarkRun` ID they share, so
results from separate executions stay distinguishable.

### Creating Test Runs

Expand Down
4 changes: 3 additions & 1 deletion configs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,4 +101,6 @@ All configurations track key metrics:
- **Resource Usage**: CPU, memory, disk I/O
- **Throughput**: Transactions per second, gas per second

For detailed metric analysis, view the interactive dashboard after running benchmarks.
Every metric is written to the output directory as JSON. See
[../docs/report-data-contract.md](../docs/report-data-contract.md) for the
layout and field definitions.
20 changes: 10 additions & 10 deletions docs/report-data-contract.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Report data contract

This document describes what the `report/` web UI in this repo expects
from whatever produces benchmark data. The intent is to let `base/benchmark`
become a **pure visualization tool** — any runner (Go, Rust, future
monorepo) can produce data the report consumes, as long as it follows
this contract.
This document describes what the benchmark report UI expects from whatever
produces benchmark data. The UI itself lives in [`base/ui`](https://github.com/base/ui)
under `app/benchmark/`; this repo is one of the producers. Any runner (Go,
Rust, future monorepo) can produce data the report consumes, as long as it
follows this contract.

The current Go runner in `runner/` is one producer; the future Rust
runner in the Coinbase monorepo will be another. Both write the same
Expand Down Expand Up @@ -191,7 +191,7 @@ on-the-wire shape.

The set of metric keys inside `ExecutionMetrics` is open. The
frontend has a registry of "known" metrics with units and labels at
`report/src/metricDefinitions.ts`; anything not listed there still
`base/ui/app/benchmark/metricDefinitions.ts`; anything not listed there still
renders, just with raw key names.

## Comparison groups (automatic)
Expand Down Expand Up @@ -294,7 +294,7 @@ A natural follow-on is to drop the merged `metadata.json` response
entirely and serve per-run JSON via an index endpoint
(`GET /api/v1/runs?since=<ts>` paginated). The frontend already has a
data-source abstraction in
`report/src/services/dataService.ts` — swapping out the metadata
`base/ui/app/benchmark/services/dataService.ts` — swapping out the metadata
fetch would be a single function change. But this requires a
frontend change, so it's a bigger lift than the producer-side
upgrades above.
Expand Down Expand Up @@ -336,11 +336,11 @@ UI silently breaks.
- Comparison synthesizer:
`protocols/base-benchmarking/report-api/internal/services/comparison.go`
- Frontend types (what the UI expects):
`report/src/types.ts::BenchmarkRun`
`base/ui/app/benchmark/types.ts::BenchmarkRun`
- Frontend metric definitions (keys/units/labels):
`report/src/metricDefinitions.ts`
`base/ui/app/benchmark/metricDefinitions.ts`
- Frontend filter logic (how dropdowns are auto-discovered):
`report/src/hooks/useBenchmarkFilters.ts` and `report/src/filter.ts`
`base/ui/app/benchmark/hooks/useBenchmarkFilters.ts` and `base/ui/app/benchmark/filter.ts`
- Current Go runner that produces this shape:
`runner/benchmark/result_metadata.go::Run` and
`runner/service.go::applyClientVersion`
Loading
Loading