Skip to content

feat(agent): collect and store host diagnostic bundles - #20

Open
memetics19 wants to merge 23 commits into
mainfrom
feat/agent-diagnostics
Open

feat(agent): collect and store host diagnostic bundles#20
memetics19 wants to merge 23 commits into
mainfrom
feat/agent-diagnostics

Conversation

@memetics19

@memetics19 memetics19 commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Summary

Adds read-only diagnostic bundles. pulse-agent --diagnose collects evidence about why a host failed — OOM kills, disk usage, busiest processes, container and systemd state, Proxmox guest status, plus logs for whatever is already broken — and Pulse stores it and serves it back. The goal is answering "why did this break" on a remote host without opening an SSH session.

Collection is on demand in this PR. Triggering it automatically from an incident is follow-up work.

Type of change

  • feat — new feature
  • fix — bug fix
  • refactor — no behaviour change
  • docs
  • ci / build
  • test
  • chore
  • Breaking change

Changes

agent

  • New internal/diagnostics package: collectors for dmesg OOM kills (global and memory-cgroup forms), df usage, ps top processes, docker ps -a, systemctl list-units --failed, and qm list.
  • Collectors take an injected Runner, so they are tested against recorded command output rather than the host the tests run on. Each command is individually timeout-bounded, and a timeout is reported as one rather than as signal: killed.
  • Sections degrade independently: a host that denies dmesg, has no Docker, or is not a Proxmox node still produces a useful bundle, with unavailable collectors recording their own error. A failing command's own output is folded into that error, so cannot connect to the docker daemon survives instead of a bare exit status.
  • Captures the recent journal for units already failed and recent output for containers already stopped. Targets come from what the host reports as broken, never from the server. Bounded to 200 lines, 5 targets per section, 32 KiB each.
  • Full-filesystem detection suppresses only filesystems with no backing store. tmpfs, overlay, and loop-backed mounts are flagged, because hiding a genuinely full filesystem is worse than a spurious flag on a read-only image.
  • --diagnose flag for one-shot collection. Prints the bundle with no server configured — the only mode available when Pulse itself is unreachable — and uploads it when --server and --token are both given. Supplying only one is rejected rather than silently falling back. An interrupted run prints its partial bundle but exits non-zero.
  • Collection carries no shared deadline: every command is already bounded, and a shared budget only let one slow phase starve later ones. The upload takes its own bounded child of the signal context, so it has a finite budget of its own and Ctrl-C still cancels.

api

  • Migration 11 adds agent_diagnostics. A bundle is owned by the agent that produced it and carries no other association; it cascades only from that agent.
  • POST /api/ingest/diagnostics, authenticated with the same agent bearer token as metric ingest. The bundle must be a JSON object; its section contents are not validated, because the agent owns that schema and the payload is stored verbatim so collectors can change without a server-side migration.
  • GET /api/agents/{agentID}/diagnostics returns an agent's recent bundles, newest first, behind the existing agents:read scope. Reading a host's evidence is an admin action, not something the agent's own ingest token can do.
  • Bundles fall under the existing retention_days window and are removed by the retention worker.

docs

  • New docs/diagnostics.md, added to the nav; architecture.md links to it.

chore

  • Per-module coverage.* artifacts added to .gitignore.

Breaking changes

None. Migration 11 is unreleased and was rewritten in place rather than superseded, so no released schema or API changed.

Test plan

  • cd api && go test ./... -count=1
  • cd agent && go test -race ./... -count=1
  • cd cli && go test ./... -count=1
  • cd ui && npx tsc --noEmit (UI unchanged)
  • gofmt -l and go vet ./... clean in all three modules
  • End-to-end through the real router: an agent pushes a bundle with its own token, an operator reads it back with an agents:read API key.
  • Manual: pulse-agent --diagnose on macOS. disk and docker collected; kernel, processes, systemd, and proxmox degraded to per-section errors as designed, and the bundle stayed usable.

Running it live also caught a bug the unit tests had not: /dev was flagged as a full disk, because pseudo filesystems permanently report 100%. Covered by a regression test.

Known gaps

  • The agent binary is not shipped by the release or install path. release.yml cross-compiles only api/cmd/pulse, and install.sh never mentions the agent, so pulse-agent is reachable only by building from source. Pre-existing, but this PR is the first change that depends on it. Worth a follow-up before this feature is usable by anyone who did not clone the repo.
  • No recorded fixtures captured from a real Proxmox host. The qm list, systemctl, and dmesg parsers are tested against representative output. They should be checked against a live host before being relied on.
  • ps -eo ... --sort=-pcpu is procps-specific, so the processes section degrades on macOS and cannot be smoke-tested there. Reading the kernel ring buffer needs root; the kernel section degrades when the agent runs unprivileged.

Adds a diagnostics package that gathers evidence about why a host failed:
OOM kills from dmesg, filesystem usage, the busiest processes, container
and systemd unit state, and Proxmox guest status.

Collectors take an injected Runner, so they are tested against recorded
command output rather than the host the tests run on. Each command is
individually timeout-bounded, since a wedged host is exactly when
diagnostics matter most.

Sections degrade independently: a host that denies dmesg, has no Docker,
or is not a Proxmox node still produces a useful bundle, with the
unavailable collectors recording their own error.

Pseudo filesystems are listed but never reported full — they permanently
report 100% capacity, and a false "disk full" would drive a wrong
diagnosis.
Adds migration 11 with an incident_diagnostics table and a
POST /api/ingest/diagnostics endpoint authenticated with the same agent
bearer token as metric ingest.

The bundle is stored verbatim as JSON: the agent owns the bundle schema,
so collectors can be added or changed without a matching server-side
migration.

incident_id is optional. An on-demand bundle describes a host at a moment
in time and need not belong to an incident; when set, diagnostics cascade
with the incident.

collected_at records receipt time so ordering stays consistent across
agents with skewed clocks. The agent's own timestamp survives inside the
stored payload.
Collects a single diagnostic bundle and prints it. When --server and
--token are supplied it uploads the bundle as well.

Printing works with no server configured at all, which is the only mode
available when Pulse itself is unreachable — the case where a bundle is
most needed.

Extracts the shared postJSON helper from Push so both ingest paths use
one request path.

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review decision: Request changes

The fixed-command design, per-section degradation, authenticated agent ownership, and test seams are solid. I found seven actionable correctness/lifecycle issues in the inline comments. The highest-risk path is a degraded host: sequential timeouts can consume the full parent deadline, and the configured upload then starts with an expired context.

I also verified the three-dot PR diff with git diff --check, gofmt, go vet ./..., full tests, and race-enabled tests in agent, api, and cli; all passed. Current GitHub checks are green. One important coverage gap remains: .github/workflows/ci.yml vets/tests api but not the newly changed agent module. Please add the agent module to CI and include blocking-runner, real stderr, malformed-output, CLI flag/exit, bundle-shape, and retention tests with the fixes.

Comment thread agent/internal/diagnostics/collect.go
Comment thread agent/internal/diagnostics/exec.go Outdated
Comment thread agent/internal/diagnostics/disk.go Outdated
Comment thread api/internal/db/queries/diagnostics.sql Outdated
Comment thread api/internal/handlers/diagnostics.go Outdated
Comment thread agent/cmd/agent/main.go
Comment thread agent/internal/diagnostics/docker.go Outdated
An OOM kill says a service died; the log says why. The agent now pulls
the recent journal for units already in the failed state and recent output
for containers already stopped.

Log targets are derived from what the host reports as broken, never taken
from the server, so this adds no new input the server can influence.

Capture is bounded so a bundle stays under the server's 1 MiB request cap:
the last 200 lines, at most 5 targets per section, truncated to 32 KiB
each. Truncation keeps the tail, where the failure is.

Also folds a failing command's own output into its section error. A bare
"exit status 1" is useless in a tool whose whole job is explaining
failures — "cannot connect to the docker daemon" and "usage: sudo dmesg"
are the actual diagnoses.
… credentials

Collection could consume the whole diagnose deadline, so PushDiagnostics
inherited an expired context and the upload failed on exactly the degraded
hosts this feature targets. The upload now runs on a context detached from
collection; Pusher still bounds the request with its own client timeout.
A wedged host therefore yields a partial bundle that still reaches Pulse.

Suppressing tmpfs and overlay hid real incidents: a full tmpfs is
memory-backed exhaustion and a full overlay is a container's writable layer
filling up. Suppression is now limited to mounts that read 100% by design.
Because df -P names the device rather than the filesystem type, read-only
image mounts are matched by /dev/loop* — a snap squashfs was being flagged
as a full disk.

Supplying only one of --server or --token silently fell back to local-only
mode and exited 0, so automation could believe evidence reached the server
when it never did. It now fails with a message.
PruneIncidentDiagnostics was generated but never called, so bundles
accumulated without bound while the pruner trimmed only check results.
Diagnostics now fall under the same retention window.

The ingest handler accepted any non-empty JSON, storing null, numbers,
strings, and arrays as diagnostic evidence. It now requires a JSON object.
Section contents stay unvalidated — the agent owns that schema.
A killed process surfaces as "signal: killed", which does not tell an
operator the command hit its time limit — the likeliest failure on the
wedged hosts this feature exists to diagnose.

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review decision: Request changes

I re-reviewed the full merge-base diff and the fixes added after the prior review. The credential pairing, failed-command stderr, non-object bundle rejection, tmpfs/overlay handling, and retention-pruner issues are genuinely improved. The remaining blockers are resource bounds, phase-budget isolation, cancellation correctness, ingest admission control, and incident ownership.

Additional release evidence is still missing: the green Go check runs only the API module, the agent binary is not shipped by the release/install path, and no recorded real Proxmox fixture demonstrates parser compatibility. The handler also still accepts {}/objects without the minimum sections envelope and does not reject a trailing JSON document.

Local verification at this exact head: git diff --check, gofmt, go vet ./..., and go test -race ./... -count=1 passed independently in agent, api, and cli. golangci-lint is not installed and the repository has no .golangci.yml, so no golangci-lint claim is made.

Comment thread agent/internal/diagnostics/exec.go Outdated
Comment thread agent/internal/diagnostics/exec.go
Comment thread agent/internal/diagnostics/docker.go
Comment thread agent/cmd/agent/diagnose.go Outdated
Comment thread agent/internal/diagnostics/disk.go Outdated
Comment thread api/internal/db/migrations/11_incident_diagnostics.up.sql Outdated
Comment thread api/internal/handlers/diagnostics.go Outdated
…e budget

ExecRunner reported a cancelled caller as "timed out after Ns" because it
inspected the derived context, which inherits parent cancellation. Ctrl-C
now reports cancellation, and only a real command timeout reports a timeout.

The 60s whole-run deadline is removed. Every command is already bounded by
commandTimeout, and the shared budget only let one slow phase starve later
ones: five docker-logs calls could consume it before systemd and proxmox
collection ran, so optional log enrichment could erase primary evidence.

The upload no longer uses context.WithoutCancel, which removed operator
cancellation along with the deadline. It takes its own bounded child of the
signal context instead, so it gets a fresh finite budget and Ctrl-C still
aborts the run.

Loop devices are no longer suppressed when flagging full filesystems.
Writable loop-mounted ext4/XFS is common and df -P cannot distinguish it
from a read-only image, so the rule hid genuinely full filesystems. A
spurious flag on a snap mount is the lesser failure.
Nothing ever populated incident_id: PushDiagnostics always sent a bundle
alone, so the column, the request field, the nullable foreign key and the
cascade were all built for a caller that does not exist.

It was also unsound. Any authenticated agent could claim any incident,
there is no agent-to-incident relation to authorize the claim, and the
cascade let that unverified link delete evidence the agent owned.

The table is renamed agent_diagnostics, which is what it holds: evidence a
specific agent collected about its own host. A server-owned, validated
association can be added when a real caller for it exists.

Migration 11 is unreleased, so it is rewritten in place rather than
superseded. No released schema or API changes.

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review decision: Request changes

The latest commits resolve several earlier blockers: diagnostic evidence is now agent-owned, the unvalidated incident link is gone, upload retains caller cancellation with its own finite budget, parent cancellation is no longer mislabeled as a local timeout, retention is wired, and loop-backed filesystems are no longer blanket-suppressed.

I still would not merge this feature yet. The new line comments identify four reproducible correctness/CLI defects plus two design/documentation gaps. The most serious are that the OOM parser misses the normal memory-cgroup form used for container kills, SIGTERM can still produce a partial local bundle and exit 0, and removing the shared deadline expands the sequential worst case to about 160 seconds of collection before the 30-second upload window.

Two earlier resource concerns also remain unresolved: CombinedOutput buffers complete command output before any truncation, and the final bundle has no aggregate cap before it reaches the server's 1 MiB request limit. Malformed numeric fields also still become plausible zero values. These should be covered by huge-output, aggregate-size, malformed-fixture, slow-runner, and signal/subprocess tests.

Delivery evidence is incomplete: the repository CI and release workflow gate the API but do not test, vet, format-check, build, install, or package pulse-agent; no recorded Proxmox fixtures exercise the advertised target. The PR description is also stale relative to the now-wired pruning and removed incident association.

Fresh verification at head 2119616: all module tests passed, go vet ./... passed in agent/api/cli, changed diff check passed, and go test -race ./cmd/agent ./internal/diagnostics ./internal/pusher -count=1 passed. The public CLI and parser probes described inline still reproduce. GitHub cannot record a formal CHANGES_REQUESTED review from the PR author's account, so this is submitted as a comment review with the decision stated explicitly.

Comment thread agent/internal/diagnostics/kernel.go Outdated
Comment thread agent/cmd/agent/main.go
Comment thread agent/cmd/agent/diagnose.go
Comment thread agent/cmd/agent/main.go
Comment thread docs/diagnostics.md Outdated
Comment thread api/internal/db/queries/diagnostics.sql
The OOM regex matched only the global form, so a container killed by its
own cgroup memory limit — the primary Docker failure mode — produced an
empty kernel section for exactly the case it exists to explain. The match
is now case-insensitive, covering "Memory cgroup out of memory: Killed
process" as well.

An interrupted diagnosis also reported success: Collect records
"context canceled" in the sections it could not reach, but runDiagnose
returned nil and the CLI exited 0, so a script would treat a partial
cancellation bundle as a completed run. The partial bundle is still
printed and the exit status now reflects the interruption.
Bundles could be uploaded but not read back: no handler, worker, or CLI
retrieved them, so the remote path wrote and pruned evidence while
delivering no capability. Reaching it meant opening the database, which
defeats the reason for pushing evidence off the host.

GET /api/agents/{agentID}/diagnostics returns an agent's recent bundles,
newest first, behind the existing agents:read scope — reading a host's
evidence is an admin action, not something the agent's own ingest token
can do. The payload is returned as JSON rather than a quoted string so
callers parse it once. The limit defaults to 10 and caps at 50, because
bundles are large.

Covered end to end through the real router: an agent pushes with its own
token, an operator reads it back with an API key.

Also corrects the disk documentation, which still described the /dev/loop
suppression removed earlier.
@memetics19
memetics19 force-pushed the feat/agent-diagnostics branch from 328da14 to 828250a Compare August 19, 2026 11:52
@memetics19

Copy link
Copy Markdown
Owner Author

Review rounds 1–3: all 20 line threads addressed and resolved

Every inline thread across the three reviews now has a reply and is resolved. Summarising here, since the review-level decisions are not threads and cannot be resolved individually.

Fixed — 15 of 20

Area Change
OOM parser Case-insensitive match, so memory-cgroup container kills are caught, not just the global form
Upload lifecycle Own bounded child of the signal context; no WithoutCancel, cancellation preserved
Phase budgets Shared 60s deadline removed, so log enrichment can no longer starve primary sections
Cancellation Caller cancellation no longer mislabelled as a local command timeout
Exit status An interrupted run prints its partial bundle and exits non-zero
Command errors A failing command's own stderr survives instead of a bare exit status 1; timeouts say so
Disk tmpfs, overlay, and loop-backed mounts flagged; only backing-store-less filesystems suppressed
Incident link Removed entirely — column, request field, FK, and cascade; table renamed agent_diagnostics
Retention PruneAgentDiagnostics wired into the retention worker
Bundle validation Non-object bundles rejected
Credentials Half-configured --server/--token rejected instead of silently exiting 0
Retrieval GET /api/agents/{agentID}/diagnostics behind agents:read — stored bundles are now readable
Docs Disk suppression behaviour corrected to match the implementation

Declined — 5, each with reasoning in its thread

  • Bounded writers / aggregate cap. Every command is bounded at source (-n 200, --tail 200, kernel ring buffer), and captured logs are truncated to 32 KiB × 5 targets per section, giving a ~320 KiB ceiling under the 1 MiB cap that MaxBody already enforces.
  • Whole-collection deadline. A shared budget is what produced the starvation flagged in round two. The ~160s worst case needs every one of sixteen commands to hang fully, and Ctrl-C now works and exits non-zero. The automated collection path is where an explicit bound belongs, designed with enrichment separated from primary work.
  • Per-agent rate limiting. Single-tenant self-hosted; the token is one the operator minted and installed on their own host. MaxBody caps each request and retention caps lifetime.
  • Parser completeness contract. docker ps --format '{{json .}}' and friends are machine-generated. Malformed numeric fields do fall back to zero, but Full is computed from the capacity column, so a zero elsewhere is cosmetic rather than a false diagnosis. If it ever bites, the version with teeth is one guard — non-empty output parsing to zero rows becomes a section error — not a cross-parser contract.
  • Positional-argument rejection. A stray argument still yields exactly the diagnosis requested; nothing is silently misconfigured. Three lines if wanted.

On the delivery points

  • CI is no longer API-only. ci: validate PR titles and run all three Go modules #22 merged and pull_request workflows run from the merge ref, so this PR's Go job already runs vet and test across api, agent, and cli — visible as ##[group]vet agent / ##[group]test agent in the job log.
  • The release and install path is a real, open gap. release.yml cross-compiles only api/cmd/pulse and install.sh never mentions the agent, so pulse-agent ships to nobody. It predates this PR but this is the first change that depends on it, and it is now recorded in the description. Best handled as its own PR rather than widening this one.
  • No recorded Proxmox fixtures. Accurate, and called out in the description from the start. The qm list, systemctl, and dmesg parsers are tested against representative output and should be validated against a live host. That is the one claim here unit tests genuinely cannot settle.

Description refreshed — it previously still listed the pruning gap and the incident_id 500, both of which no longer exist.

One verification caveat worth stating rather than glossing: the exit-0-on-SIGTERM fix is proven at the function seam, not through a live signal. Collection completes in well under a second on macOS, so the signal always landed after the run.

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. I re-reviewed the full merge-base diff at 828250a after the latest fixes. The incident-ownership, cancellation, credential-pairing, cgroup-OOM, filesystem-classification, retention-wiring, and write-only-lifecycle findings are resolved. Four fresh defects remain inline: the history route retroactively grants sensitive evidence access to existing agents:read keys; the count-only history limit can materialize roughly 50 MiB of payload twice in heap; command timeouts do not terminate descendants holding output pipes; and df output is labeled kilobytes without forcing 1 KiB units.

The earlier aggregate-upload/output-cap concern remains unresolved rather than repeated inline: ten legal 32 KiB logs can exceed the 1 MiB request limit after JSON escaping, and inventories are not aggregate-capped. The existing discussion already contains the requested fix.

Missing release evidence: CI and release workflows still vet/test only api/, and releases build only pulse, not pulse-agent. There is no cross-binary production-adapter contract test, legacy-scope denial test, maximum history-response test, descendant-process timeout test, or recorded Proxmox fixture gate.

Local verification at this head: all three Go modules passed go test ./... and go vet ./...; targeted agent and API packages passed race testing; gofmt and git diff --check were clean. golangci-lint is not configured/installed, so that check could not be run. GitHub reports all configured checks green, but those checks do not exercise the new agent module.

Comment thread api/internal/server/server.go
Comment thread api/internal/handlers/diagnostics.go Outdated
Comment thread agent/internal/diagnostics/exec.go Outdated
Comment thread agent/internal/diagnostics/disk.go Outdated
…meouts

POSIX df -P reports 512-byte blocks on BSD and under POSIXLY_CORRECT, and a
probe on a real host printed a 512-blocks header, so available_kb was
silently double the true value. -k forces 1024-byte blocks. Verified
against the host: the collector and df -Pk now agree.

exec.CommandContext kills only the direct child, so a descendant that
inherited the pipe kept CombinedOutput blocked past the deadline — a
100ms timeout returned after 2.01s in a reproduction. That breaks the
no-hang guarantee on exactly the wedged hosts this package targets.
Cmd.WaitDelay bounds that wait and forces the pipes closed.
Diagnostic bundles carry journal entries, container logs, process names
and filesystem paths. Serving them under the existing agents:read scope
meant every key already issued for agent inventory silently gained access
to them — a backward-compatible privilege expansion rather than just a new
route. The history route now requires an explicit diagnostics:read grant,
exposed in key management and documented alongside the other scopes.

The history limit is also cut from 50 to 5, defaulting to 3. A bundle can
approach the 1 MiB ingest cap, so fifty rows would materialise tens of
megabytes twice over — once by the driver, again as raw JSON — while
holding the single SQLite connection. Reading recent evidence needs a
handful of bundles, not a page.
@memetics19
memetics19 force-pushed the feat/agent-diagnostics branch from 21d5adf to d7a93b9 Compare August 19, 2026 12:47

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. The two newest commits genuinely resolve the prior df-unit, inherited-pipe return-latency, least-privilege vocabulary, and large-history-count findings. This fresh pass found one merge-blocking authorization bypass and two additional correctness/lifecycle defects: the registered trailing-slash route falls back to agents:read; WaitDelay returns while leaving the descendant process alive; and the existing retention setting now deletes diagnostic evidence although the admin UI still describes monitor/check history only. Two smaller comments tighten the user-facing df contract and the new cap regression test.

Fresh verification at head d7a93b9: go test ./... and go vet ./... passed in agent, api, and cli; targeted race tests passed for the changed agent and API packages; gofmt and git diff --check were clean. Focused real-router and PID-liveness regressions fail exactly on the two blockers described inline. golangci-lint is unavailable and no repository configuration exists. GitHub's configured checks are green.

Comment thread api/internal/middleware/apikey.go
Comment thread agent/internal/diagnostics/exec.go
Comment thread api/internal/worker/pruner/pruner.go
Comment thread docs/diagnostics.md Outdated
Comment thread api/internal/handlers/diagnostics_test.go Outdated
Chi serves a route with and without its trailing slash, but the scope check
matched on a path suffix, so "/api/agents/1/diagnostics/" did not end in
"/diagnostics" and fell through to the generic /api/agents rule. An
agents:read key received 200 on that spelling while correctly getting 403
on the other — the privilege boundary added in the previous commit was one
character from being bypassed.

requiredScope now normalises the trailing slash before matching, which
fixes the whole class rather than the diagnostics path alone. The scope
test exercises both spellings through the real router.

Also tightens the history limit test: asserting "at most five" would have
passed if the endpoint returned nothing, so it now requires exact counts
for the clamped, default, and under-limit cases.
Wiring diagnostics into the existing retention worker quietly widened what
"retention_days" deletes. The admin labelled it "Keep monitor history for"
and the docs described check history only, so an operator lowering it for
uptime storage would have discarded diagnostic evidence without knowing.
The label and the retention docs now name both data classes.

The disk collector also runs df -Pk rather than df -P, because POSIX -P
alone reports 512-byte blocks on BSD and under POSIXLY_CORRECT and would
make available_kb double the true value. The source table said df -P; the
flag is part of the correctness contract, so it is now documented as one.

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. Fresh full review of the three-dot diff from merge base 8bfb0d7 through ce844fa found four reproducible evidence-contract defects. The latest commits correctly fix equivalent-path authorization, exact history-limit coverage, retention disclosure, and df unit documentation. Repository tests, race tests, vet, formatting, diff checks, and current GitHub checks are green; focused regression probes reproduce each issue below.

Comment thread agent/cmd/agent/diagnose.go Outdated
Comment thread agent/internal/pusher/diagnostics.go
Comment thread agent/internal/diagnostics/exec.go
Comment thread agent/internal/diagnostics/kernel.go
exec.CommandContext signals only the direct child, so a descendant kept
running after the timeout and repeated diagnoses would accumulate orphans
on an already-degraded host. Commands now run in their own process group
and the group is killed on cancellation, with WaitDelay retained for pipe
cleanup. A regression test asserts the descendant PID is actually dead.
The Unix implementation is build-tagged, with a no-op elsewhere.

Docker output that parses to nothing is no longer reported as a healthy
empty section. Docker prints nothing at all when there are no containers,
so text that yields no rows means the section is unreliable and says so.

The rendered bundle is capped at 900 KiB. Oversized bundles previously
failed as an opaque 400 from the server's 1 MiB request cap; they now fail
locally with a message naming the cause, after the local copy is printed
so the evidence is never lost.

A stray positional argument is now rejected rather than silently ignored.
The release workflow cross-compiled only the server, so pulse-agent —
which runs on the hosts being monitored, not the Pulse host — was
reachable only by building from source. All three workspace binaries are
now built for linux and darwin on amd64 and arm64, checksummed together,
and attached to the release. Documented with an install snippet.

Diagnostic uploads are also rate limited to one per agent every five
seconds, returning 429 with Retry-After. Bundles approach the 1 MiB
request cap, so a cron misfiring in a tight loop could outpace retention.
The interval is short enough not to impede a human running --diagnose
twice.
…shed

The previous assertion probed the descendant with kill(pid, 0), which
still succeeds while a killed process lingers as a zombie awaiting reaping.
Reaping timing differs by platform, so the test passed on macOS and failed
on the Linux CI runner.

It now has the descendant append to a file and asserts the file stops
growing once the command is cancelled, which is the behaviour that matters
and is independent of zombie semantics. Verified to fail when the process
group setup is removed.
…dlines

A local write failure returned before PushDiagnostics, so a full disk or a
broken stdout redirect — the exact degraded-host cases this command serves
— destroyed the only off-host copy. Rendering and delivery are now
independent effects and both errors are joined, so the upload is attempted
regardless and nothing is silently swallowed.

The Pusher's shared 10s HTTP client timeout also capped every request,
silently overriding the 30s budget runDiagnose grants an upload. The client
carries no timeout now and each operation bounds itself through the
context: metrics keep a short self-imposed limit because they run on a
ticker with no caller deadline, while a bundle upload uses the budget it
was given.

Command timeouts also discarded whatever the command had already written.
A process that explains itself on stderr and then hangs is the most useful
kind of failure, so that output is now carried into the timeout error under
the same size bound as ordinary failures.

OOM kills retain the kernel's own timestamp. Without it a kill from a
previous boot is indistinguishable from one in the current incident, which
turns evidence into a red herring.

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision: request changes. The collection and transport fixes from the previous round are solid, but this head introduces one reproduced admission bug and leaves the release lifecycle incomplete for the binaries it now publishes. Fresh verification cross-compiled all 12 artifacts; vet passed in api/agent/cli; focused changed-area tests and agent race tests passed. The full API suite still exposes its pre-existing timing-sensitive TestLoginRateLimited failure when the ten bcrypt attempts take long enough for token refill.

Comment thread api/internal/handlers/diagnostics.go Outdated
Comment thread .github/workflows/release.yml
Comment thread .github/workflows/release.yml
Comment thread agent/internal/diagnostics/procgroup_test.go Outdated
Comment thread docs/diagnostics.md
Comment thread docs/diagnostics.md Outdated
The rate limit recorded an agent's slot before decoding and inserting, so a
malformed body returned 400 and then locked out the corrected retry behind
a 429. A failed insert masked itself the same way. Admission is now checked
without recording and committed only once the bundle is stored.

The interval is a guard against a misfiring loop, not a storage ceiling;
retention is what bounds total size, and the comment now says so rather
than implying otherwise.

The descendant in the process-group test also loops a bounded number of
times. If the group kill regresses the assertion fails, and an unbounded
loop would then keep writing forever on a developer machine or a
persistent runner.
The release workflow verified only the api module, so a tag could publish
pulse-agent or pulse-cli while that module's tests were red. Cross
compiling proves the binary builds, not that it works. The pre-release job
now vets and tests api, agent, and cli, matching what CI does on pull
requests.

Publishing pulse-cli also did not make it usable: deploy/install.sh sets up
the server only, while getting-started immediately invokes pulse-cli for
the Uptime Kuma import, so a user following the documented path reached
command not found. Both pulse-cli and pulse-agent now have documented,
checksum-verified download steps.

Also drops a reference to an agent systemd unit the repository does not
ship, and corrects an API example that used a limit the endpoint clamps.

@memetics19 memetics19 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision: request changes. The two newest commits correctly ensure rejected/failed uploads do not consume the cooldown and gate release packaging on tests for all three Go modules. A fresh full three-dot review at 47feae0 still found four actionable issues inline: a deterministic same-agent admission race, a successful-parent WaitDelay path that leaves the process group alive, checksum instructions that do not fail closed, and a remaining systemd documentation contradiction. No Critical findings.

Verification at this exact head: configured GitHub checks are green; git diff --check and gofmt are clean; go vet passed in api, agent, and cli; full agent and CLI tests passed; focused changed API tests and diagnostic race tests passed. A bounded public-handler probe produced two 204 responses for concurrent same-agent uploads in every run. A bounded process probe returned after about 2.31 seconds and observed its descendant continue after return in 3/3 runs. The full API suite also hit the unchanged timing-sensitive TestLoginRateLimited failure after bcrypt work allowed token refill; that is a pre-existing test caveat, not a finding against this diff. The PR description remains stale where it says the read route uses agents:read; code and docs correctly require diagnostics:read.

return
}

if !h.diagnosticAllowed(agent.ID, time.Now()) {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Make upload admission atomic through persistence. diagnosticAllowed releases the mutex here, while recordDiagnostic reacquires it only after decode and insert, so two simultaneous valid requests for the same agent can both observe an empty slot. A barrier request body reproduced two 204 responses in every run, defeating the stated runaway-loop/storage guard. Decode and validate first, then atomically reserve a per-agent in-flight slot; commit its timestamp only after insertion and release it on insertion failure. Add a public-handler regression requiring exactly one 204 and one 429, plus a failed-insert retry case.

// the pipe would otherwise keep CombinedOutput blocked well past the
// deadline, defeating the timeout on exactly the wedged hosts this package
// exists for. WaitDelay bounds that wait and forces the pipes closed.
cmd.WaitDelay = waitDelay

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Terminate the process group when WaitDelay wins after the direct child exits successfully. Cmd.Cancel runs when the context fires, but a parent can exit zero while a descendant retains the output pipe; CombinedOutput then returns exec.ErrWaitDelay before the 5-second context deadline and no group kill occurs. A bounded probe returned after about 2.31 seconds and the descendant's marker kept growing afterward in 3/3 runs. Detect errors.Is(err, exec.ErrWaitDelay) and explicitly terminate the group through the platform helper before returning; test a parent that backgrounds a bounded writer and exits immediately, then assert no post-return activity.

Comment thread docs/getting-started.md

curl -fsSLo pulse-cli "${BASE}/pulse-cli_${OS}_${ARCH}"
curl -fsSLo SHA256SUMS "${BASE}/SHA256SUMS"
grep " pulse-cli_${OS}_${ARCH}\$" SHA256SUMS | sed "s|pulse-cli_${OS}_${ARCH}|pulse-cli|" | shasum -a 256 -c -

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Make checksum verification gate installation. These are separate shell commands, so a checksum mismatch (or missing shasum) returns non-zero but does not stop the following chmod && sudo mv; an unverified binary is still installed. The identical pattern appears in docs/diagnostics.md:96. Chain verification and placement with fail-closed control flow, select sha256sum when available with shasum -a 256 as fallback (matching deploy/install.sh), and abort when neither tool exists.

Comment thread docs/diagnostics.md
"error": "dmesg: exit status 1: usage: sudo dmesg"
```

Reading the kernel ring buffer requires root on most systems. Run the agent as

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Remove the remaining reference to a shipped systemd unit. This sentence still says the unit runs as root, while the new note later in this page says Pulse does not ship an agent unit. The previous reply claimed this phrase was removed, but it remains at the current head; say only that root is recommended and describe the degraded kernel section.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant