feat: coverage-replayer — minimal mainnet block set maximizing mega-evm branch coverage

.gitignore

@@ -6,4 +6,10 @@
 .mcp.json
 
 # database
-validator-data
+validator-data
+
+# coverage-replayer run data & artifacts (transferred out-of-band, never via git)
+/data/
+*.profraw
+*.pid
+*.log

AGENTS.md

@@ -5,7 +5,7 @@ This file provides guidance to AI agents (e.g., Claude Code, Codex, Cursor, etc.
 ## Project Overview
 
 Stateless validator for MegaETH — validates blocks using SALT witness data without requiring full chain state.
-The workspace contains two binaries: `stateless-validator` (chain-following validator) and `debug-trace-server` (RPC server for debug/trace methods).
+The workspace contains three binaries: `stateless-validator` (chain-following validator), `debug-trace-server` (RPC server for debug/trace methods), and `coverage-replayer` (offline tool that derives a small mainnet block set reproducing all the mega-evm and revm execution coverage a chain scan observed).
 See `README.md` for detailed documentation and quickstart.
 
 ## Build & Development Commands
@@ -43,6 +43,7 @@ The project uses nightly `2026-02-03` toolchain (edition 2024, rust-version 1.95
 | `stateless-r2`         | `crates/stateless-r2`         | Shared R2 witness primitives: SigV4 signer, object-key layout, endpoint parsing, signed PUT, and the retrying witness-object GET fetcher over either the signed S3 API or an unsigned Cloudflare custom domain; consumed by mega-reth's uploaders (write) and both binaries' R2 witness sources (read) |
 | `stateless-validator`  | `bin/stateless-validator`     | Main binary: chain sync, parallel validation workers (`app.rs` / `runner.rs` / `main.rs`)                                                                                                |
 | `debug-trace-server`   | `bin/debug-trace-server`      | Standalone RPC server for debug/trace methods                                                                                                                                            |
+| `coverage-replayer`    | `bin/coverage-replayer`       | Offline coverage tool: replays blocks under LLVM branch instrumentation (`backfill`, by range or `--blocks-file`), dedups per-block coverage bitmaps (evaluated llvm-cov regions and branch arms) into patterns, and computes a greedy covering block set (`set-cover` / `report` / `inspect`). `inspect --dump-pool` exports the candidate block pool that carries a scan across a mega-evm bump. The measured scope is the mega-evm checkout plus the revm execution crates in `measured-crates.txt`; the instrumented `[profile.coverage]` build goes through `cov-rustc-wrapper.sh`, which instruments only that scope and the workspace |
 
 Additional directories: `test_data/` (integration test fixtures including genesis config), `audits/` (security audit reports).
 
@@ -143,18 +144,18 @@ In local cache mode with a `--witness-generator-endpoint` plus at least one fall
 With the `--r2-*` flag group (endpoint, bucket, access key id, secret), every request-serving witness fetch tries a direct SigV4-signed R2 GET (light decode, capped at half the remaining witness budget) before the RPC chain, falling back on any failure; `--r2-max-concurrent-requests` caps R2 GETs separately from the RPC witness semaphore.
 `--r2-custom-domain` is the alternative R2 target (mutually exclusive with `--r2-endpoint`, rejected at startup by name): unsigned GETs of `/{key}` through a Cloudflare custom domain fronting the bucket, which negotiates HTTP/2 (many in-flight GETs multiplex over a few connections instead of holding one each against the h1.1-only S3 endpoint) and can serve the immutable witness objects from edge cache; optional `--r2-access-client-id`/`--r2-access-client-secret` attach Cloudflare Access service-token headers (rejected at startup on a non-loopback `http://` domain, and validated as header values there so a stray newline fails by name instead of becoming a per-GET retryable transport error).
 Leftover S3 flags alongside the domain are rejected by name rather than silently ignored, the client sends a `User-Agent` (Cloudflare's Browser Integrity Check 403s requests without one), and the configured target is published as `debug_trace_r2_target_info{target}` / `r2_target_info{target}` so the target-less R2 series can be attributed during a rollout.
-Every `--r2-*` coherence rule — empty values, target exclusion, leftovers, an incomplete S3 quad, the Access pair, the connection count, and tuning flags with no target — lives in `stateless_common::validate_r2_flags`, so both binaries give the same verdict in the same words; each error names the offending flag, which clap cannot do without its `error-context` feature.
+Every `--r2-*` coherence rule — empty values, target exclusion, leftovers, an incomplete S3 quad, the Access pair, the connection count, and tuning flags with no target — lives in `stateless_common::validate_r2_flags`, so both binaries give the same verdict in the same words; each error names the offending flag and what is wrong with its value.
 Version selection on the custom domain is pure ALPN (no `http2_prior_knowledge`, so the plaintext loopback path keeps working), which means a grey-clouded record, a non-Cloudflare origin, or a zone with HTTP/2 off degrades to HTTP/1.1 while the h2 tuning goes inert: the fetcher warns once with the protocol it actually got and publishes `..._r2_negotiated_http_version_info{version}`, and `pool_max_idle_per_host` is bounded so the h1.1 fallback cannot accumulate idle sockets that `pool_idle_timeout(None)` would never reap.
 One `reqwest::Client` holds exactly one HTTP/2 connection and hyper never opens a second to relieve a saturated one (a pooled h2 connection reports liveness rather than stream capacity, and its dispatch channel is unbounded), so the edge's per-connection stream limit — Cloudflare advertises 100 in the `SETTINGS_MAX_CONCURRENT_STREAMS` it sends on every connection (`CLOUDFLARE_MAX_CONCURRENT_STREAMS`; `nghttp -nv https://<domain>/` reads what a given zone offers) — is a per-process ceiling rather than a per-request one.
 `--r2-connections` (default 1) is what lifts it: it holds that many clients and picks one *per attempt*, so a retry leaves the connection that just failed, and one dropped connection no longer takes every in-flight GET down with it — the availability argument, and on the validator the difference between a blip and a whole in-flight window falling back to the RPC gateway at once.
 The pick is work-conserving: a connection with a free permit, searched from a rotating cursor, so a GET is never queued behind a connection whose permits are held by a slow transfer while another sits idle, and one budget of `max` is not silently partitioned into `N` budgets of `max/N` (which queues distinctly worse at the same offered load). Only when every connection is full does a fetch wait, and it waits on the cursor's own pick rather than on whichever has the most room — under saturation that one is the connection that just dropped every GET riding it.
 `--r2-max-concurrent-requests` stays the cap across all of them, split evenly and rounded up (rounding down would leave some connection at zero permits and wedge every GET routed to it), so raising the connection count alone spreads the same concurrency thinner instead of raising the ceiling; the per-connection share is what must stay at or below the stream limit, and the fetcher warns at startup when it exceeds it.
 The count is published as `debug_trace_r2_connections` / `r2_connections`, and is rejected by name at zero, on a non-numeric or blank value, on the S3 target (HTTP/1.1 already opens a socket per in-flight GET there), and above the cap it divides — more connections than permits would leave some of them permanently idle.
-It travels as text and is parsed after clap, so a blank env line — what a templated env file renders for a variable a role does not set — is named rather than aborting startup through clap's unnamed value error.
+It travels as text and is parsed after clap, so a blank env line — what a templated env file renders for a variable a role does not set — is diagnosed by the R2 rules like every other blank `--r2-*` value, rather than aborting startup in clap's parser before those rules run.
 The validator splits the two caps the same way: `--r2-max-concurrent-requests` caps R2 GETs while `--witness-max-concurrent-requests` sizes only the RPC witness path, so a budget written for one service cannot silently become the other's.
 **Neither binary has a witness-source mode flag: configuring an R2 target *is* the switch.** With one configured the validator tries the bucket before its `--witness-endpoint` chain, exactly as the trace server does, and any R2 failure hands that one block to RPC (`r2_witness.rs` — a 3-attempt budget and no pacing pause, since the block's next stop is that chain rather than a blind re-enqueue); with no `--r2-*` flag set, witnesses come from RPC alone.
 `--witness-endpoint` is therefore always required on the validator, and every `--r2-*` rule runs on every startup, so a half-configured target, a blank value or an orphaned tuning flag is named rather than read as "no R2 configured" and silently downgraded to the RPC path.
-"Blank value" covers every `--r2-*` flag that travels as text — target, credentials, Access pair, connection count — which is why `--r2-connections` is an `Option<String>`; the two numeric tuning flags (`--r2-max-concurrent-requests`, `--r2-connect-timeout-ms`) are parsed by clap instead, so a blank one aborts before the rules run, with clap's unnamed "invalid value for one of the arguments".
+"Blank value" covers every `--r2-*` flag that travels as text — target, credentials, Access pair, connection count — which is why `--r2-connections` is an `Option<String>`; the two numeric tuning flags (`--r2-max-concurrent-requests`, `--r2-connect-timeout-ms`) are parsed by clap instead, so a blank one aborts in clap's parser before the rules run (clap's error names the flag and the value it could not parse).
 The orphan rule is judged from the value the rules already read, so an RPC-only validator that inherits `STATELESS_VALIDATOR_R2_MAX_CONCURRENT_REQUESTS` from a shared template now fails to boot where that variable used to be inert.
 Carrying only the pre-split `--witness-max-concurrent-requests` into an R2 deployment warns instead: R2 is left uncapped, which the fetcher cannot flag on its own, since with no cap there is no per-connection share to compare against the edge's stream limit.
 The whole fast path per block is bounded by one `--rpc-per-attempt-timeout-ms`, permit wait included, because the attempt count alone does not bound it: an endpoint that accepts connections and then stalls spends a full per-attempt timeout on each of the three tries, and blocks queued behind the concurrency cap wait through several such holders — a brownout absorbed far too slowly to keep the pipeline moving. A healthy fetch is sub-second, so the budget only ever bites on a stall.
@@ -208,7 +209,7 @@ The background chain-sync prefetch routes by freshness against the last observed
 ## Test Organization
 
 Unit tests are embedded in source files alongside the code they test.
-Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, compression, consistency, performance, timing_header) and in `bin/stateless-validator/tests/integration.rs` (CLI parsing, mock-RPC pipeline, mainnet single-block validation).
+Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, compression, consistency, performance, timing_header), in `bin/stateless-validator/tests/integration.rs` (CLI parsing, mock-RPC pipeline, mainnet single-block validation), and in `bin/coverage-replayer/tests/` (`replay_fixtures.rs`: worker replay glue over the `test_data/mainnet` fixtures; `worker_protocol.rs`: the worker subprocess's stdout protocol).
 Test data (block JSON files, contract bytecode, witness data) is stored in `test_data/`.
 
 ## Version Control
@@ -280,8 +281,9 @@ When implementing a new feature or bug fix, consider these additional aspects:
     The upstream witness generator serializes `(SaltWitness, MptWitness)` with bincode legacy, then zstd-compresses, then base64-encodes, and sends as a `"v0:<base64>"` JSON-RPC string.
   - **Local DB storage** (contracts, light witnesses) uses `bincode::config::standard()` (varint encoding, more compact) with lz4 compression.
   - These two formats are **not interchangeable**. `legacy()` and `standard()` produce different binary layouts.
-- **All persistent state goes through `ValidatorDB`.**
-  Do not create separate database files or ad-hoc persistence; use the existing redb tables.
+- **All persistent validator state goes through `ValidatorDB`.**
+  Do not create separate database files or ad-hoc persistence inside the validator; use the existing redb tables.
+  Each binary owns its own store: the trace server has `ServerDB`, and `coverage-replayer` keeps an offline pattern store under its `--data-dir`.
 - **Keep documentation up to date.**
   When making changes, check whether related documentation (README, this file) needs updating.
 - **One sentence, one line.**

Cargo.toml

@@ -1,5 +1,6 @@
 [workspace]
 members = [
+    "bin/coverage-replayer",
     "bin/debug-trace-server",
     "bin/stateless-validator",
     "crates/stateless-common",
@@ -65,7 +66,7 @@ base64 = { version = "0.22", default-features = false }
 bincode = { version = "2.0", features = ["serde", "alloc"], default-features = false }
 bytes = "1.11"
 chrono = { version = "0.4", default-features = false }
-clap = { version = "4.6", features = ["derive", "env", "std"], default-features = false }
+clap = { version = "4.6", features = ["derive", "env", "error-context", "help", "std", "usage"], default-features = false }
 dashmap = { version = "6.1", default-features = false }
 dotenvy = "0.15"
 dyn-clone = "1.0"
@@ -123,3 +124,19 @@ opt-level = 3
 debug-assertions = true
 incremental = true
 debug = true
+
+# Instrumented builds for coverage-replayer: coverage only cares about "was it
+# executed", so trade peak runtime speed for much faster builds (no LTO, many
+# codegen units). Do NOT add `-C link-dead-code` (it monomorphizes dead generic
+# code and fails const-eval asserts in revm). The instrumentation flags come
+# from a rustc wrapper rather than RUSTFLAGS, so that only the measured code is
+# instrumented (see the wrapper for why that matters); the explicit --target is
+# how it tells host artifacts apart. Build with:
+#   RUSTC_WRAPPER="$PWD/bin/coverage-replayer/cov-rustc-wrapper.sh" \
+#     cargo build --profile coverage -p coverage-replayer --features coverage \
+#       --target "$(rustc -vV | sed -n 's/host: //p')"
+[profile.coverage]
+inherits = "release"
+opt-level = 2
+lto = "off"
+codegen-units = 16

README.md

@@ -27,7 +27,7 @@ The stateless approach eliminates the need for validators to run on high-end har
 
 ## Project Structure
 
-The workspace contains two binaries and five library crates:
+The workspace contains three binaries and five library crates:
 
 | Crate                  | Path                          | Purpose                                                                                                                                                                              |
 | ---------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
@@ -38,6 +38,7 @@ The workspace contains two binaries and five library crates:
 | `stateless-r2`         | `crates/stateless-r2`         | Shared R2 witness primitives: SigV4 signer, object-key layout, endpoint parsing, signed PUT, and the retrying witness-object GET fetcher over either the signed S3 API or an unsigned Cloudflare custom domain; consumed by mega-reth's witness uploaders (write), this repo's validator, and the trace server's witness source (read) |
 | `stateless-validator`  | `bin/stateless-validator`     | Main binary: chain sync, parallel validation workers                                                                                                                                 |
 | `debug-trace-server`   | `bin/debug-trace-server`      | Standalone RPC server for debug/trace methods                                                                                                                                        |
+| `coverage-replayer`    | `bin/coverage-replayer`       | Offline coverage tool: replays mainnet blocks under LLVM branch instrumentation and derives a small (greedy) block set that reproduces all the execution coverage the scan observed, of mega-evm and the revm execution engine under it |
 
 Additional directories: `test_data/` (integration test fixtures including genesis config), `audits/` (security audit reports).
 
@@ -82,7 +83,7 @@ cargo run --release --bin stateless-validator -- \
   Which of the two a run produces follows from how far behind it is: a tip-following validator fetches inside that window, so all of its misses are frontier misses — routine and numerous — and `kind="missing"` stays at zero; watch the frontier rate there instead.
   `kind="missing"` is the bucket-integrity signal during catch-up and fixed `--end-block` backfills, where blocks sit far below the head. A hole that first appears near the tip is not caught here, since the block is fetched once, falls back and is never re-probed; that belongs to whatever monitors the uploader.
   With no `--r2-*` flag set at all, witnesses come from the RPC chain alone; a half-configured target, a blank value, or a tuning flag with no target is rejected at startup by name rather than read as "no R2 configured".
-  "Blank value" covers the flags that travel as text; the two numeric tuning flags are parsed by clap, so a blank one aborts earlier with clap's unnamed error (see AGENTS.md for the full rule).
+  "Blank value" covers the flags that travel as text; the two numeric tuning flags are parsed by clap, so a blank one aborts earlier, in clap's parser (see AGENTS.md for the full rule).
   The R2 attempt for one block is bounded in total by a single `--rpc-per-attempt-timeout-ms`, permit wait included, so an endpoint that accepts connections and then stalls costs the block one upstream hop's wall clock rather than one per retry before the RPC chain takes over.
 - `--r2-custom-domain`: alternative R2 target that replaces the four flags above — unsigned HTTP/2 GETs through a Cloudflare custom domain fronting the bucket (mutually exclusive with `--r2-endpoint`, and any of the four left set is rejected at startup by name rather than silently ignored; optional `--r2-access-client-id`/`--r2-access-client-secret` attach Cloudflare Access service-token headers, which require an `https://` domain unless it is loopback; the domain's cache rule must set 404s to bypass cache, or a cached pre-upload 404 pushes those blocks onto the RPC path for the negative-cache TTL and false-fires the `kind="missing"` alarm once they age past the frontier band)
 - `--report-validation-endpoint`: RPC endpoint URL for reporting validated blocks via `mega_setValidatedBlocks` (disabled if not provided)