Skip to content

make a killed runsc say who killed it - #1154

Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:fix/ci-dump-forensics
Open

make a killed runsc say who killed it#1154
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:fix/ci-dump-forensics

Conversation

@orangeCatDeveloper

@orangeCatDeveloper NekoPunch (orangeCatDeveloper) commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Fixes #1153

A CI run where runsc was killed says neither why it died nor enough to work it out afterwards.

runsc reports signal: killed for an OOM kill, an outside SIGKILL and exec.CommandContext acting on a done context alike, and every subcommand in runsc.go dropped ctx.Err(), the one thing that separates them. Meanwhile the dump tailed each pod at 300 lines and never looked at termination state, node conditions, events or dmesg, so the line that would have explained a failure had usually scrolled off.

ctxErrorSuffix names the context state next to the failed command, and stays silent when the error already carries it — which is what Run returns if the context died before Start. It says what was observed rather than naming a killer: a done context makes cancellation the strong candidate, but the process could still have been killed for another reason at the same moment.

On the CI side the workers now dump untailed, plus their previous container's log and describe, while the control plane keeps its tail: it runs the length of the job, and its history is not where a failing actor's evidence is. dmesg is grepped for OOM rather than tailed, since the kill line scrolls off a busy ring buffer, with a plain tail kept alongside for anything the grep misses.

Reading the two together: a context error points at cancellation, a dmesg record confirms an OOM, and neither means the cause is still open.

Before and after, for the three ways the subprocess dies:

scenario                            before                after
context deadline expired            signal: killed        signal: killed (context: context deadline exceeded)
context cancelled                   signal: killed        signal: killed (context: context canceled)
outside SIGKILL, context untouched  signal: killed        signal: killed

The dump was run against a live cluster: every section produced output, and the OOM grep found a real record a tail -100 would have missed.

Memory cgroup out of memory: Killed process 1133445 (prometheus) total-vm:3625916kB, anon-rss:1039984kB

hack/verify-all.sh passes except proto-fmt, which needs a clang-format this machine does not have; no proto files are touched.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

CI dumps tailed every pod at 300 lines and never looked at termination
state, node conditions, events or dmesg, so a failure whose explanation
had scrolled off could not be diagnosed from the run. And runsc reports
`signal: killed` for an OOM kill, an outside SIGKILL and a cancelled
context alike, with nothing in the error to tell them apart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Several e2e flakes cannot be diagnosed from the failed run

1 participant