kconmon-ng

Someone says "the network is fine." Prove it.

kconmon-ng makes inter-node connectivity a measured fact. An agent on every node probes every other node over TCP, UDP and ICMP every five seconds, resolves DNS and checks HTTP endpoints from each node, and exports latency, jitter and packet loss for every ordered node pair, per protocol; a failed probe fires an MTR trace, so the bad hop is on record before anyone looks. Each protocol is measured separately for each pair, so the partial failures that cost you the evening (UDP dropping on one pair while TCP stays clean) read as exactly that instead of vanishing into a green aggregate. Everything downstream is built on those measurements: an N×N matrix, a topology map, MTR path history, incident timelines, alert rules that Prometheus evaluates, and a ?at= on the URL that rewinds every page to the minute it broke.

_{Overview mid-incident on a six-node kind cluster: worker4 has lost TCP to and from every peer, so 10 of 30 pairs fail and the worst-pairs table ranks five of them at 100.0%. Two console rules fire, NodeTcpUnreachable for worker4 and PairPmtuBlackHole for a path MTU black hole from worker3 to worker5, and three incidents are open, one per problem plus a global one for zone b.}

Matrix, TCP, live, mid-break, zoomed to 150% so the grid fills the page: worker4 is red at 100% both as a row and as a column, 10 of 30 cells. The 20 pairs among control-plane, worker, worker2, worker3 and worker5 stay green with a p95 RTT of 1.8 to 2.2 ms.	Incidents, scoped to worker3 → worker5 over 1h: 338 timeline entries, the newest the pair's TCP diagnostic timeouts between audit rows, and the Signals panel showing the fail ratio going from 0.0% to 100.0%. Its packet-loss chart sits at 0% until about 06:17 and holds 100% after.
Alerting: two console-managed rules, both synced against the cluster and enabled, the chart's own PrometheusRule offered for import as a foreign rule (1 group, 13 rules, Helm), and one declared maintenance window at the bottom.	Time Machine: the same Matrix resolved at 9/26/2026 08:37:30, about a minute and a half before that break. The amber banner marks the viewed instant; all 30 pairs are green, worker4 included, with a p95 RTT of 2.2 to 3.9 ms.
Bundled Grafana Overview on a healthy stand: the key indicators sit on top (agents, leader, pairs, failures, and path MTU black holes and reduced paths), then the worst-pair and MTR bars, with the charts and tables below. Everything reads 0 failing across 30 pairs.	Bundled Zone Heatmap during a staged break, with worker2 in zone a and worker5, zone c's only node, cut off: zone-to-zone UDP loss, p95 UDP and ICMP RTT, and TCP and UDP failure ratios over the last 15 minutes, plus MTR traces triggered over the last hour. Zone c reads about 69% to 70% as a source and as a destination on loss and both failure ratios, zone a to itself about 46%, zones a and b to each other about 23%, zone b to itself 0.0%, and the RTT tables stay green because only answered probes have a round trip.
Bundled Node Detail dashboard for kc-accept-worker3: 5 peers probed outbound, every worst-case failure and UDP loss tile at 0.00%, no MTR traces from this node, and per-peer outbound charts with p95 UDP RTT flat at about 480 µs.

Documentation

Everything lives at https://esdmitrii.github.io/kconmon-ng/; start with:

• Install in 15 minutes
• Enable the console
• Console guide
• Helm values
• Metrics and alerting
• External agents
• FAQ
• Release notes

What it does

• Measures every pair on every protocol. One controller Deployment, one agent per node; agents register over gRPC and get a live-pushed peer list, and every metric carries source_zone/destination_zone read from the node labels. TCP, UDP, ICMP and DNS run on a 5s interval; HTTP is opt-in. Architecture
• Traces the failure while it is happening. A failed TCP, UDP or ICMP probe triggers MTR for that pair under a per-pair cooldown; paths are content-hashed at ingest, so path history is a list of changes, not identical traces. Routes (MTR)
• Finds MTU black holes. A once-a-minute path MTU probe sends full-size datagrams with DF set and bisects on loss, so a pair where handshakes and pings pass but large packets vanish goes red with the size that still crosses, instead of staying green on every other plane. Catch an MTU black hole
• Speaks Prometheus. Stable metric names, three Grafana dashboards in dashboards/, fourteen alert rules with the chart (UDP loss, failing TCP, an MTU black hole per pair and per zone pair, a node unreachable or isolated, DNS and external checks, a pair gone silent, two zone rules, KconmonAgentsMissing and KconmonControllerDown, so a monitor that goes quiet pages you, plus an opt-in KconmonExternalAgentDown for bare-host agents), each tunable via prometheusRule.<alertName>.*; the console's own families share the prefix, documented by their HELP strings. Metrics and alerting
• Adds an optional Console, off by default. N×N heatmap, topology map, curated charts; an Investigate page merging nine timeline sources, causes ranked by documented arithmetic; alert rules from typed templates or raw PromQL, reconciled into one PrometheusRule; diagnostics, external targets, schedules; four auth modes, 26 permissions across four built-in roles, an audit log, HMAC-signed webhooks. Read-only pages need only a Prometheus URL; the rest needs PostgreSQL. Console guide
• Rewinds. ?at= on any URL resolves every read surface through that instant, with mutations disabled while it is engaged. Time Machine
• Takes vantage points outside the cluster. Since v2.3.0 the same agent installs on a bare host from deb or rpm and joins through a TLS gateway with a bearer token and optional client-certificate pinning. External agents

Quick start

Kubernetes 1.31+, Helm 4 (or Helm 3 from 3.14; the chart is an OCI artifact) and a Prometheus to scrape. The Operator is optional; with it you get the bundled ServiceMonitor and alert rules:

helm upgrade --install kconmon-ng oci://ghcr.io/esdmitrii/charts/kconmon-ng \
  --set serviceMonitor.enabled=true \
  --set prometheusRule.enabled=true

# one controller pod plus one agent per node
kubectl get pods -l app.kubernetes.io/name=kconmon-ng -o wide

Metrics flow within seconds; then follow Install in 15 minutes for the first queries, the dashboards, the scrape job without the operator and the make local-up Minikube demo. The agent needs no added capabilities and passes restricted PSS unchanged (why).

No Prometheus Operator?

Moved to the site: one endpoints-role scrape job covers the fleet, and the alert rules are plain PromQL.

Turn on the Console

Moved to the site: Enable the console, one flag on the same release, then one capability at a time.

From your terminal

kubectl-kconmon reaches the controller's HTTP API over a client-go port-forward. A failed check exits 2, distinct from 1 for CLI or API errors, so it composes in pipelines; -o json prints the raw result.

$ kubectl kconmon topology
NODE     ZONE         READY   AGENT                           AGENT IP
node-1   us-east-1a   yes     node-1-kconmon-ng-agent-aaaaa   10.0.0.1
node-2   us-east-1b   yes     node-2-kconmon-ng-agent-bbbbb   10.0.0.2
node-3   us-east-1c   no      -                               -

$ kubectl kconmon check node-1 node-2 --type udp
OK udp node-1 -> node-2 (us-east-1a -> us-east-1b)  duration=1.1ms
  sent=5 recv=5 loss=0% rtt=1.1ms jitter=240µs

Install it from the krew index with kubectl krew install kconmon, or pin a release with kubectl krew install --manifest-url https://github.com/EsDmitrii/kconmon-ng/releases/latest/download/kconmon.yaml.

Scope and limits

• External agents are supported since v2.3.0: TLS gateway on the controller, bearer token, optional client-certificate pinning, deb and rpm packages. 2.4.0 adds per-agent ports (an external host no longer mirrors the fleet's port pair), a Prometheus HTTP SD endpoint so external hosts are scraped without hand-written targets, agent.hostNetwork for probing from the node's own address, and console awareness of external members.
• What stays out: Windows hosts (why), two agents on one IP, a CSR flow for agent certificates; the full list is What v1 does not do.
• Scale envelope: 50–100 nodes on a full mesh is production-proven; each directed pair keeps 78 active series and pairs grow as N×(N−1), about 770k series at 100 nodes. topology.mode: sparse (since v2.3.0) trims probed pairs to a ring plus cross-zone chords. Arithmetic and levers: Scaling and cardinality.

How it compares

kconmon-ng inherits its name and shape from kconmon by Karl Stoney (TypeScript on Node.js, archived in June 2026) and is a ground-up Go implementation of the same idea, not a fork, extended with ICMP, reactive MTR tracing and the Console. Reach for goldpinger when an HTTP-level pod graph is enough, for kubenurse when the question is the path through ingress and API server, and for kconmon-ng when you need per-protocol pair evidence with the bad hop already traced; the feature table is in the FAQ.

Links

• Documentation: https://esdmitrii.github.io/kconmon-ng/
• Chart: Artifact Hub, oci://ghcr.io/esdmitrii/charts/kconmon-ng; images ghcr.io/esdmitrii/kconmon-ng-{agent,controller,console} on GHCR
• Releases (deb/rpm host packages, krew manifest), dashboards/, CONTRIBUTING.md, SECURITY.md

Development

make build      # agent, controller and console binaries → bin/
make test       # unit tests; make test-race adds the race detector
make lint       # golangci-lint; make helm-lint runs the chart against CI value sets
make local-up   # Minikube + Prometheus + Grafana + kconmon-ng, one command

CI runs lint (the e2e suite included), race tests, cross-compile, helm-lint and a build of every image on every PR; a v* tag builds and signs the images, runs e2e on them, and only then publishes the chart, the release and, on the newest stable tag, :latest. Start with CONTRIBUTING.md and hack/README.md.

Community

Governance, maintainers, roadmap, code of conduct and adopters. Running kconmon-ng somewhere? A pull request adding a line to ADOPTERS.md helps more than a star.

License

Apache License 2.0. See LICENSE.