ublk churn wedge repro

Root cause found, October 2026: the wedge is an io_uring worker-pool race, and this program now reproduces it. Upstream 38aa434ab933 ("io_uring/io-wq: add exit-on-idle state") and 912146614894 ("io_uring: allow io-wq workers to exit when unused"), in 7.0 and backported to stable 6.18.11, mark a thread's io-wq exit-on-idle about 7.5 ms after the thread closes its last ring. A control command the thread punts while one of its idle workers is exiting is handed to that worker and never runs. Upstream fixed it in ad41f325be ("io_uring/io-wq: don't strand work when a worker exits"); ublk-go avoids it by running control commands on threads that keep a ring open (e2b-dev/ublk-go#241).

The program never wedged before because it started a fresh thread per device, and a fresh thread has no exiting worker to strand. Reproduce with one reused control thread and a random pause with no ring open between devices, and stop the device before tearing the server down:

REPRO_REUSE_THREAD=1 REPRO_GAP_MS=15 REPRO_STOP_FIRST=1 ./ublk_churn_repro 2000 20

Measured in virtme-ng microVMs with each guest's vCPUs pinned to one host CPU: with this command 10/10 runs wedge at ADD_DEV on 6.18.48 and 0/20 with the two commits reverted; gap mode with the default teardown wedged 20/20 at ADD_DEV on 7.2.6 as well. Without REPRO_STOP_FIRST a separate teardown hang also appears, on every kernel including the reverted one: the kernel's asynchronous partition scan (6.18.4+) is still reading through the new disk when the worker is cancelled, and STOP_DEV's del_gendisk waits for the scan's open_mutex forever. Stopping the device while the worker still serves IO (ublksrv's order) removes it: 0/20 on the reverted kernel.

Other modes: REPRO_KICK_ON_STALL=1 submits an IOSQE_ASYNC NOP after a 2 s stall and reports whether the stranded command completes; REPRO_HOLD_ON_WEDGE=1 keeps the process alive at a wedge so its threads and stacks can be inspected. The text below is the August record, kept as it was.

Attempted reproducer for a ublk_drv control-command hang seen on Ubuntu's HWE 7.0 kernel:

7.0.0-28-generic (linux-image-generic-hwe-24.04, noble)
a ublk control command never completes; the caller blocks forever in
io_uring_enter and every later ublk control command wedges behind it

What the bug actually is

Diagnosed later, from the Go server rather than from this program, so the framing below (and the original "START_DEV never completes") is narrower than the truth. Captured hangs hit ADD_DEV as well as START_DEV — whichever punted control command happens to be in flight.

At a hang, no thread is inside ublk code at all. The io_uring tracepoints show the whole story: io_uring_submit_req for the command, io_uring_queue_async_work two microseconds later, then silence. Linux 7.0 punts every non-read-only ublk control command to io-wq (ublk_ctrl_uring_cmd_may_sleep returns -EAGAIN under IO_URING_F_NONBLOCK), and the punted work is never picked up: no worker runs it, no completion is posted, and the process's only io-wq worker belongs to a different task and sits idle in io_wq_worker. So the fault is in io_uring's io-wq, not in ublk_drv, which merely arranges for the punt.

Affected releases, measured with TestChurnLiveness (ten runs per kernel, ~250 create/close cycles each): 6.17 and 6.18.0 clean, 6.18.40 hangs, 7.0.0 hangs 5 times in 40 runs, 7.0.14 hangs 5 in 10. It is the stable point releases that are affected, not the series bases — so something landed in the 6.18.y range and is still in 7.0.y.

Why this program never reproduced it

Two server-side differences remain, and one of them was removed from this program on purpose:

1. Fixed-file registration. ublk-go used to register the char fd and submit FETCH/COMMIT with IOSQE_FIXED_FILE; dropping it took 7.0.0 from 29 hangs in 40 runs to 5. This program tried it and reverted (see the note in worker_fn) because the registration's reference to the char device outlives close(cfd) and deadlocked DEL_DEV at iteration 0 on every kernel. Reintroducing it with ublk-go's teardown ordering is the untested variant.
2. Concurrency. This loop is strictly serial. In ublk-go the parallel integration suite hangs on 7.0 nearly every run, while serial churn needs thousands of cycles — so concurrent device creation is by far the stronger trigger, and it is the ingredient this program lacks entirely.

Neither was pursued: ublk-go reproduces the hang directly and its tracepoint capture is better evidence than a C loop that passes.

The program is a miniature ublk server built on raw io_uring (IORING_OP_URING_CMD, SQE128). It does not depend on ublk-go or liburing. Each iteration is one full device lifecycle:

1. UBLK_U_CMD_ADD_DEV
2. UBLK_U_CMD_SET_PARAMS
3. a worker thread submits UBLK_U_IO_FETCH_REQ for every tag and then blocks in io_uring_enter(GETEVENTS), servicing IOs with UBLK_U_IO_COMMIT_AND_FETCH_REQ
4. UBLK_U_CMD_START_DEV
5. cancel the worker, close the char fd, UBLK_U_CMD_STOP_DEV, UBLK_U_CMD_DEL_DEV

Two details are load-bearing and worth knowing before you simplify it:

• Two threads are required. A URING_CMD completion is task work bound to the submitting task, so a single-threaded version deadlocks by construction: the thread issuing START_DEV is the only one that could run the FETCH task work the kernel is waiting for.
• The worker must actually service IOs. START_DEV calls add_disk(), whose partition scan reads the new disk; those reads must be completed by the worker before START_DEV returns. A worker that only waits deadlocks the kernel in blk_add_partitions → read_part_sector.

A per-iteration alarm() watchdog turns the hang into a non-zero exit naming the iteration and phase, so CI reports a failure instead of timing out silently.

Run

sudo modprobe ublk_drv
make
sudo ./ublk_churn_repro            # 5000 iterations, 30s watchdog
sudo ./ublk_churn_repro 20000 30   # iterations, watchdog seconds

Exit codes: 0 all iterations completed, 1 a command returned an error, 3 the watchdog fired (the wedge).

Status: this reproducer never triggered the bug

It runs 3000 iterations clean on every kernel tested, including ones where the Go server it was reduced from (https://github.com/e2b-dev/ublk-go, TestChurnLiveness) hangs within tens of iterations. Matching the Go server's queue count, queue depth, max IO size and device flags did not change that, and neither did finally testing the two releases that do hang — v6.18.40 and v7.0.14 were added to CI in the last commit and both passed 3000 iterations. That rules out "wrong kernel" as the explanation; see "Why this program never reproduced it" above for what is left.

Run it with udevd stopped. udev opens each freshly added /dev/ublkbN to probe it, and if that probe is in flight at teardown, del_gendisk waits for udev's opener while the opener waits for IO no server is left to serve — deadlocking STOP_DEV on every kernel, unaffected ones included. ci/guest.sh stops udevd for that reason.

Known results

C reproducer, udevd stopped, 3000 iterations each:

Kernel	Result
7.0.0-28-generic (noble HWE)	passes
mainline v7.0, v7.0.12	passes
6.14 (noble HWE series)	passes
6.17.0-7-generic	passes (also 1500 iterations locally, ~21/s)
6.8.0-137-generic (noble GA)	passes, but see note
mainline v6.18.40	passes — and this is a kernel ublk-go hangs on
mainline v7.0.14	passes — and this is a kernel ublk-go hangs on

The last two rows are the decisive ones. Until they were added, every kernel tested here was one where the hang is mild or absent, because the mainline tag picker's sort -V ranked v6.18/ above v6.18.40/ and silently pinned the leg to the series base release. With that fixed, this program still passes 3000 iterations on both.

For contrast, TestChurnLiveness in the Go server, run alone and serialized, ten repeats: mainline 6.18.40 wedged 4 of 6 (iterations 77–200) and 7.0.0-28-generic wedged 6 of 6 (iterations 12–248).

Note on 6.8: this loop passes, but 6.8 is separately affected by CVE-2025-37906 (ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd, fixed upstream in v6.15, never backported to noble's 6.8 GA kernel), which panics that kernel under close-under-load with in-flight IO. That is a different bug from the one reproduced here.

See REPORT.md for the bug-report text.

CI

.github/workflows/kernels.yml runs the reproducer across kernels in a virtme-ng microVM on a ubuntu-24.04 runner, plus directly on the ubuntu-26.04 runner's own 6.18 host kernel. Because a wedge can take the guest down, each leg relies on the watchdog exit rather than the job timeout.