Skip to content

Kill the compiler's process group on timeout - #344

Merged
maleadt merged 1 commit into
mainfrom
ao/compiler-group-kill
Nov 5, 2026
Merged

maleadt merged 1 commit into
mainfrom
ao/compiler-group-kill

Conversation

@AntonOresten

@AntonOresten AntonOresten commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

tileiras runs ptxas. When the compiler timeout fired, run_and_collect SIGKILLed tileiras alone, and ptxas kept running on its own for however long the kernel takes it to compile, burning a core and memory after we had already reported a TileCompilerTimeoutError.

Outside of Windows, the compiler now runs in a session of its own (detach), and a timeout kills its whole process group. In its own session the compiler no longer receives the terminal's Ctrl-C, so an interrupted wait kills the group too. The kill holds the I/O lock like kill(::Process) does, so the process can't be reaped (and its pid reused) between the check and the signal.

On Windows, detach would also take the compiler out of libuv's job object that kills it when Julia exits, so there nothing changes: only the compiler is killed.

tileiras leaves its /var/tmp/mlir-kernels-* temporaries behind when killed, with SIGKILL or SIGTERM alike; that's not addressed here.

@AntonOresten
AntonOresten marked this pull request as draft October 29, 2026 14:00
@github-actions

Copy link
Copy Markdown

Documentation preview: https://juliagpu.github.io/cuTile.jl/previews/PR344/

@maleadt
maleadt force-pushed the ao/compiler-group-kill branch from e900407 to 52facbc Compare November 5, 2026 07:49
@maleadt
maleadt marked this pull request as ready for review November 5, 2026 07:49
@maleadt

maleadt commented Nov 5, 2026

Copy link
Copy Markdown
Member

Rebased and slightly reworked. The fix stands, but tileiras sends ptxas output to a log file, so the actual problem was the orphaned ptxas, not a blocked read. I also dropped detach on Windows, where it would take tileiras out of libuv's kill-on-exit job object.

`tileiras` runs `ptxas`. On a timeout, `run_and_collect` killed `tileiras`
alone, leaving `ptxas` running unattended for however long the kernel takes
it to compile.

Outside of Windows the compiler now runs in a session of its own (`detach`),
and a timeout kills its process group. Since the compiler then no longer
receives the terminal's Ctrl-C, an interrupted wait kills the group too. On
Windows, `detach` would take the compiler out of the job object that kills
it when Julia exits, so there only the compiler is killed, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZLLYWFZYfnGcR3TSUNK4A
@maleadt
maleadt force-pushed the ao/compiler-group-kill branch from 52facbc to 15dc9e6 Compare November 5, 2026 07:51
@maleadt
maleadt merged commit 0e26ca6 into main Nov 5, 2026
7 checks passed
@maleadt
maleadt deleted the ao/compiler-group-kill branch November 5, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants