Skip to content

fix: Log when the Python backend stub exits on a crash signal - #461

Open
yinggeh wants to merge 2 commits into
mainfrom
yinggeh/tri-2006-fail-qa-when-the-python-backend-stub-exits-on-a-signal
Open

yinggeh wants to merge 2 commits into
mainfrom
yinggeh/tri-2006-fail-qa-when-the-python-backend-stub-exits-on-a-signal

Conversation

@yinggeh

@yinggeh yinggeh commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What does the PR do?

Log an error when the Python backend stub exits on SIGSEGV, SIGABRT, SIGBUS, SIGILL, or SIGFPE during the graceful finalize wait. SIGKILL stays unlogged because an unhealthy stub is terminated that way on purpose.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • build
  • ci
  • docs
  • feat
  • fix
  • perf
  • refactor
  • revert
  • style
  • test

Related PRs:

Where should the reviewer start?

src/stub_launcher.cc, WaitForStubProcessWithTimeout.

Test plan:

Build the devel image from this branch and run L0_backend_python--base and L0_infer--base with the matching server branch. Intentional stub kills in model_readiness and restart must stay green.

  • CI Pipeline ID:

Caveats:

A crash already reaped by StubActive() has no status left, so it is not logged.

Background

The graceful finalize wait discarded waitpid status, so a stub that died on a crash signal looked like a normal exit.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: none

The graceful finalize wait discarded waitpid status, so a stub that died
on SIGSEGV, SIGABRT, SIGBUS, SIGILL, or SIGFPE looked like a normal exit.

Signed-off-by: Yingge He <yinggeh@nvidia.com>
@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous logging bug is fixed.

Summary

Logs when the Python backend stub exits on a crash signal during the graceful finalize wait.

  • The graceful wait logs when the stub dies from a crash signal.

Reviews (2) · Last reviewed commit: "fix: Log a stub crash before clearing it..." · Reviewed by Greptile

Comment thread src/stub_launcher.cc Outdated
The waitpid check cleared stub_pid_ before comparing it with the return
value, so the crash log never ran.

Signed-off-by: Yingge He <yinggeh@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug fix (fix: PRs)

Development

Successfully merging this pull request may close these issues.

1 participant