Repository navigation
feat: let agents read a testnode's metrics with litro metrics - #61
Merged
Merged
Conversation
….mjs Start-container discovery parsed `docker ps` text: a regex over Ports (which missed collapsed port ranges, then backtracked badly enough for CodeQL to flag it), Status text for health, and a fallback that matched unlabeled `arbitrum-testnode-<variant>` names, which caught containers litro never started. It now lists containers by the start label and reads `docker inspect` JSON: labels, state, health and HostConfig.PortBindings, which stopped containers keep too. - packages/testnode/src/containers.mjs owns everything docker-side about a container (runDocker, inspect, logs, diagnostics, discovery, selectStartContainer), which takes runtime.mjs from 1013 to 813 lines. - packages/testnode/src/endpoints.mjs is the one table of what each container port serves; the RPC URLs, the feed URLs and discovery all build URLs from it. - The CLI's checkout lookup returns core's InitContext, so stop, status and logs stop rebuilding the config dir, compose file and project name. - stop, logs and feed throw on failure, so the CLI exits non-zero. - The CLI bundle builds into apps/cli/bundle, apart from tsc's dist, which used to overwrite it; `pnpm pack:cli <dir>` builds, packs and smoke-tests the tarball for both CI and the release job. Release fixes from v0.4.0: - release.mjs tagged with a plain `git tag`, which fails under tag.gpgSign=true after the release commit was already made. It now tags lightweight as every earlier release was, drops its commit if tagging still fails, checks main matches origin/main before pushing, and moves the README's action pins (except lines marked `# pinned`). A test runs it in a temporary repo with tag.gpgSign=true. - The README's action examples moved from v0.2.10/v0.2.6 to v0.4.0. - The integration suite resolves @arbitrum/testnode from source, so it runs in an unbuilt checkout; Verify Published Image no longer needs a build step.
… from start --metrics `litro start --metrics` turns on everything the testnode can report and opens Grafana with dashboards for it; `--pprof` adds profiling. `litro stop` removes the stack, `litro status` lists its URLs. - Nitro: --metrics on the L2 sequencer, L2 validator and L3 node (docker/observability.sh), about 950-990 metric families each, published on :6070/:6072/:6074; pprof on :6071/:6073/:6075. - docker/chain-exporter.mjs: every chain from the L1 up over JSON-RPC, since Anvil has no metrics of its own. Blocks, transactions, gas, base fee and txpool under ethereum-metrics-exporter's eth_exe_* names; per rollup the batches posted, assertions created and confirmed, ArbOS version, and the delayed inbox by message kind with what users are still waiting on. - The Fast Feed relay serves GET /metrics: clients, frames, refusals by status, closes by code, upstream reconnects. Never anything derived from API keys. - Prometheus v3.15.0 and Grafana 13.2.3 run beside the testnode on their own network, configured with docker cp, so no bind mounts are needed. - Four dashboards, generated by metrics/build-dashboards.mjs, every panel captioned: Litro chains (home), Nitro nodes, Fast Feed relay, and Nitro: every metric. Proving the dashboards on a live l3-eth chain changed what they show: - Every batch appends a batch posting report to the delayed inbox, which the next batch reads, so "delivered minus read" sat at 1 on a healthy chain and read as a stuck deposit. The exporter now classifies MessageDelivered by kind and reports only user messages waiting, and how long the oldest waited. - Nitro does not implement txpool_status and logged an error per call; the exporter asks once. - Head block age, batch poster wallet, pending parent-chain transactions, assertion errors and the delayed-inbox rate were reworded or rebuilt after the live data showed how each read. - `stop` could not remove the metrics network while the testnode container was attached; it now detaches it first. test/fixtures/nitro-metrics-v3.12.1.txt holds the 999 metric families the three live nodes exported; a test fails if a dashboard queries anything not in it, the chain exporter or the relay. packages/core/test/integration/metrics.test.ts drives the chain (L2 transactions, a batch, an L1 deposit, a refused Fast Feed client) and checks Prometheus saw each one; CI runs it in test-metrics-start.
…ctions correctly A panel-by-panel review of the dashboards against a loaded chain: - Transactions per second counted ArbOS's internal start-of-block transaction, about doubling the L2 and L3 rates. The exporter reads each block's receipts (eth_getBlockReceipts) and counts only user transactions: 10 sent, 10 counted. It also counts reverted transactions and records when each chain last included a user transaction. - The relay counts frames carrying a user transaction apart from the start-of-block ones, so its rate matches the L2's, and records delivery latency (sequencer timestamp_ms to send) as a histogram. - New panels: failed transactions per second, last transaction per Arbitrum chain (head block age only for the L1, which mines every second), Fast Feed delivery latency. - Removed panels that only repeat another or stay flat on litro: blocks per second, base fee, txpool, head block gas used, cumulative assertions and batches, head block and latest confirmed assertion per node, batches waiting, sequencer and transaction feed clients, added delay, bytes per second. - Alert-only signals (queue peak, dropped transactions, posting failures, deposits waiting) are stats; the validator's assertion retries and the sequencer's static L1 price sit in a collapsed Diagnostics row; stacked panels show only series that moved; the validator no longer reports a second batch poster wallet; pending parent-chain transactions are clamped at zero; refusal and close panels show a zero baseline; time series start at zero. Nitro: every metric defaults to arb_sequencer_.*. Live on l3-eth (10/10 integration scenarios, now including a reverting transaction counted as failed and Fast Feed delivery latency measured).
`litro metrics` reads the testnode's Prometheus back as JSON, so an agent can check health or look back over a run without Grafana. Each subcommand is also an MCP tool and a skill. - `summary`: one object answering the dashboards' questions, per chain (RPC up, head, transactions and failures per second, time since the last transaction, ArbOS version), per rollup (batches, deposits waiting, assertion ages), per Nitro node (blocks behind the sequencer, CPU, memory) and for the Fast Feed relay (clients, p50/p99 delivery latency, refusals). NaN and Inf come back as null. - `list --filter <regex>`: every metric with its type, help text and the sources that report it. - `query <promql> [--since 15m --step 1m]`: current value per series, or a time series over the window. Prometheus errors are passed on. `start --keep-history` keeps 30 days of samples in a named volume (<container>-prometheus-history) that survives `stop`; without it Prometheus keeps one day and `stop` removes it. The dashboard check also holds the summary's queries to known metric names, and the metrics integration test asserts the summary for every chain, rollup and node.
…ions Review follow-ups on `litro metrics`: - metrics-questions.mjs holds each question's PromQL once. The dashboards and `litro metrics summary` both build on it, so an agent and Grafana read the same number: the summary had drifted to 1m transaction rates and a 5m latency window where the dashboards use 30s and 1m. The regenerated dashboard JSON is byte-identical. - SUMMARY is grouped by section, so field names are object keys and each section's label is stated once. That removes the per-entry section/by pairs, the summaryFields helper and its `any`, and the test that guarded against duplicate fields. - `--keep-history`'s volume carries the stack's label. `stop` says it kept it, and `stop --forget-history` removes it; litro no longer leaves a volume only `docker volume rm` can clean up. - A non-JSON answer from Prometheus (a 502 page) reports its HTTP status instead of a JSON parse error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
litro metricsreads a running testnode's Prometheus back as JSON, so an agent can check its health or look back over a run without opening Grafana. Each subcommand is also an MCP tool (metrics_summary,metrics_list,metrics_query) and a skill. Stacked on #60.summaryanswers the dashboards' questions in one object:NaN and Inf come back as
null.The summary and the dashboards share their queries.
packages/testnode/src/metrics-questions.mjsholds each question's PromQL once, so an agent and Grafana read the same number. The regenerated dashboard JSON is byte-identical.listreturns every metric with its type, help text and the sources that report it.queryreturns either the current value per series or, with--since, a time series. Prometheus's own errors are passed on.--keep-historykeeps 30 days of samples in a labelled volume,<container>-prometheus-history.stopsays it kept the volume, andstop --forget-historyremoves it. Without the flag, Prometheus keeps one day andstopremoves it.Proven on a live l3-eth chain
summarymatches the dashboards: after 6 minutes of traffic it returned all chains up, L2 at 0.9 tx/s with 0.034 failed/s, and Fast Feed p99 at 0.99 ms. The dashboards showed the same numbers.--keep-history: I started with the flag, stopped, and started again. The earliest sample stayed 1793541954.7.stop --container litro-metrics --forget-historythen left no litro containers, networks or volumes.tools/listshows the three tools, andtools/call metrics_queryreturns JSON.