nbd: eliminate queue freeze/unfreeze overhead in connection setup - #1129
Open
blktests-ci[bot] wants to merge 8 commits into
Open
nbd: eliminate queue freeze/unfreeze overhead in connection setup#1129blktests-ci[bot] wants to merge 8 commits into
blktests-ci[bot] wants to merge 8 commits into
Conversation
Author
|
Upstream branch: d58772d |
blktests-ci
Bot
force-pushed
the
linus-master_base
branch
from
August 12, 2026 13:27
29ac21d to
ea2c39d
Compare
Author
|
Upstream branch: f5bbbfe |
blktests-ci
Bot
force-pushed
the
series/1134955=>linus-master
branch
from
August 12, 2026 13:37
a62084e to
970845f
Compare
blktests-ci
Bot
force-pushed
the
linus-master_base
branch
from
August 12, 2026 18:56
ea2c39d to
4082a30
Compare
Author
|
Upstream branch: 3d6d817 |
blktests-ci
Bot
force-pushed
the
series/1134955=>linus-master
branch
from
August 12, 2026 19:25
970845f to
cae0616
Compare
blktests-ci
Bot
force-pushed
the
linus-master_base
branch
from
August 13, 2026 19:18
4082a30 to
36bd7eb
Compare
Author
|
Upstream branch: 3aa1dca |
blktests-ci
Bot
force-pushed
the
series/1134955=>linus-master
branch
from
August 13, 2026 20:03
cae0616 to
4a37f4a
Compare
blktests-ci
Bot
force-pushed
the
linus-master_base
branch
2 times, most recently
from
August 16, 2026 18:34
d89ab11 to
fdba928
Compare
Author
|
Upstream branch: fd923b3 |
blktests-ci
Bot
force-pushed
the
series/1134955=>linus-master
branch
from
August 16, 2026 20:02
4a37f4a to
2b93f2e
Compare
blktests-ci
Bot
force-pushed
the
linus-master_base
branch
from
August 17, 2026 14:10
fdba928 to
60442a3
Compare
The second conditional checking nsock->fallback_index validity is the logical inverse of the first, so drop it and let execution fall through naturally. Consolidate the two identical dev_err_ratelimited() + return paths into a single no_fallback label to reduce duplication. Reviewed-by: Yu Kuai <yukuai@fygo.io> Signed-off-by: Long Li <leo.lilong@huawei.com>
We cannot add a socket to an already running nbd device, the reconfigure for netlink can only active an inactive socket. But for ioctl path, we can call NBD_SET_SOCK after NBD_DO_IT, reject this using nbd->pid which has been setted when NBD_DO_IT. Besides, it is the root cause for commit b98e762 ("nbd: freeze the queue while we're adding connections"). Reviewed-by: Yu Kuai <yukuai@fygo.io> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
An inactive nbd device may refuse any I/O operations. The nbd_config_put function calls invalidate_disk, which sets the device capacity to zero to reject all read and write I/O. For zero-sector flush I/O requests from blkdev_issue_flush, if the write cache is disabled, the zero-sector flush I/O immediately returns 0 in submit_bio_noacct. However, since nbd_config_put does not clear the write cache state, an inactive nbd device might still have the write cache enabled. In this situation, zero-sector flush I/O will return -EIO because there is no active socket. Additionally, BLK_FEAT_FUA and BLK_FEAT_ROTATIONAL flags may also remain stale, resetting all of them ensures consistent behavior. The limits update uses queue_limits_commit_update() (the non-freezing variant) because config_refs == 0 here means every fd is closed and recv threads have drained, so no in-flight I/O can read q->limits concurrently. Reviewed-by: Yu Kuai <yukuai@fygo.io> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
nbd_add_socket() kreallocs config->socks, which a concurrent reader in nbd_handle_cmd() could UAF; commit b98e762 ("nbd: freeze the queue while we're adding connections")froze the queue to block that. But the freeze costs an RCU grace period on every socket added, and setup adds them one by one. After the previous patch, nbd_add_socket() is rejected once nbd->pid is set, so it only runs during setup. There the capacity is 0 and the write cache is off (cleared on disconnect by the preceding patch, and re-enabled only later in nbd_set_size), so submit_bio_noacct() rejects every bio before it reaches the driver -- non-zero-sector ones via bio_check_eod(), and flush-only ones via the !bdev_write_cache() branch. No I/O is in flight, so the freeze is unnecessary. Reviewed-by: Yu Kuai <yukuai@fygo.io> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Commit 242a49e ("nbd: freeze the queue for queue limits updates") added the freeze to keep in-flight commands from seeing torn queue_limits. But at startup the capacity is still 0 (invalidate_disk cleared it) and the write cache is off (the previous patch cleared it on disconnect, and nbd_set_size sets it back only after the commit), so submit_bio_noacct() rejects any bio before it reaches the driver and no I/O is in flight. Drop the freeze by checking capacity and write cache state in nbd_set_size. Reviewed-by: Yu Kuai <yukuai@fygo.io> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
The NBD_ATTR_SOCKETS walk is duplicated in nbd_genl_connect (add sockets) and nbd_genl_reconfigure (reconnect). Factor out a single helper that walks the list and calls a callback per fd; with a NULL callback it is a pure counter, used by a later patch to learn nr_hw_queues before the device exists. Returns the number of fds walked (>= 0) or a negative errno; a callback >0 will stops early. Reviewed-by: Yu Kuai <yukuai@fygo.io> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Previous commits has removed the queue freeze in nbd_add_socket and nbd_set_size during nbd device setup. However, a queue freeze can still occur when nbd_start_device calls blk_mq_update_nr_hw_queues if the socket connection count does not match nbd->tag_set->nr_hw_queues. The nbd_start_device function can be invoked through either the ioctl or netlink paths. The ioctl path only allows reusing an existing inactivate nbd device, there is nothing more we can do to prevent the queue freeze since the old nbd->tag_set->nr_hw_queues may not match the new socket connection count. Similarly, the netlink path can reuse a preferred inactivate nbd device, and again, we cannot do more in this scenario. However, the netlink path can also add a new nbd device using nbd_dev_add. In this case, we can obtain the new number of socket connections, and by adding a new argument representing the expected nr_hw_queues in nbd_dev_add, we can ensure the queue freeze is avoided for this situation. Reviewed-by: Yu Kuai <yukuai@fygo.io> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
…ices blk_mq_update_nr_hw_queues() in nbd_start_device() may cause a queue freeze. The previous commit addressed this for newly created nbd devices by setting the expected nr_hw_queues in nbd_dev_add(). However, when reusing an old inactive nbd device, the queue freeze can still occur if the old nbd->tag_set->nr_hw_queues does not match the new socket connection count. Inactive nbd devices can originate from two sources: loading the nbd module with nbds_max, which sets the default nr_hw_queues to 1, and the netlink method, which sets nr_hw_queues according to the expected number of socket connections. For the first case, add a module parameter so the default nr_hw_queues can be changed. Users who know their expected number of connections can then prevent queue freezes on pre-created devices via nbds_max. Before this patchset: real 0m2.195s user 0m0.005s sys 0m0.022s After this patchset: real 0m0.090s user 0m0.004s sys 0m0.018s Signed-off-by: Yang Erkun <yangerkun@huawei.com> Reviewed-by: Yu Kuai <yukuai@fygo.io>
Author
|
Upstream branch: 8d3ae59 |
blktests-ci
Bot
force-pushed
the
series/1134955=>linus-master
branch
from
August 17, 2026 16:51
2b93f2e to
ea1348f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: nbd: eliminate queue freeze/unfreeze overhead in connection setup
version: 8
url: https://patchwork.kernel.org/project/linux-block/list/?series=1140781