Repository navigation
馃悰 Disconnect clients that don't take a message in time, rather than queue for them - #382
Merged
Merged
Conversation
This reverts commit f8923bb. The queue of messages that each client got, and the task that writes it to the client's socket, make way for a simpler scheme, where the bus sends each message to the socket of its recipient itself, as before, but within a time limit. This brings back the sending of a982812, without such a limit yet. e64ded2, which came after the reverted commit, disconnects a client that sends a message with more file descriptors than Linux passes in one write, and builds on what the reverted commit added. It stays, and keeps what it needs of the reverted commit: * MAX_MESSAGE_FDS moves from src/peer/outgoing.rs, which goes, to src/peer/mod.rs, under the same path: busd::peer::MAX_MESSAGE_FDS. * tests/stalled_client.rs keeps the test of e64ded2, too_many_fds, along with with_bus and init_tracing, the two helpers of the reverted commit that it uses. The tests of the reverted commit go. As the test that is left runs on Linux only, the whole file is now built on Linux only, rather than each of its items, which would leave the two helpers unused elsewhere. * The rustix dev-dependency of e64ded2 stays. The serde_bytes dev-dependency, and zbus's serde_bytes feature, which only the tests of the reverted commit used, go, as does zruntime's mpmc feature. * A comment of e64ded2 had the connection of a disconnected client close once its queue is written. It now just closes. The routing and matching fixes of z-galaxy#377 and z-galaxy#378 are left as they are. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
The task that serves a client reads a message from it and forwards it before it reads the next one, and sending a message to a client only returns once the client's socket has taken all of it. So a client that stops reading held up everyone who sent to it, the bus itself included, for good. And as the bus held the read locks of the peers, of the monitors and of the names while it waited, the next client to connect waited for the write lock of the peers, and every message that the bus forwarded after that waited behind it: the bus froze. The bus still writes each message to the socket of its recipient as it forwards it, with no queue, but: * It picks the recipients of a message while it holds the locks, and only sends the message once it has let go of them. So a send that waits holds up the other messages of its sender alone. * It gives each client a time limit to take a message, SEND_TIMEOUT (1 s) by default, from the start of the send, so the limit covers the wait for the other sends to the client as well as the whole write. A client that doesn't take a message within it, or a write to which fails, gets disconnected: the bus closes its connection, which ends the client's stream of messages, and then removes the client as it does any client that goes, along with its names, be it a peer or a monitor. A message with file descriptors, for a client that didn't negotiate passing them, is dropped, as before, and the client stays. * A send that times out may leave part of its message on the socket, so nothing more is written to the client: the send flags the client before it lets go of the connection, and each send checks the flag once its turn comes. Closing the connection waits behind the send that holds it, if any, which its own time limit bounds. * A message for the bus itself has no time limit, as the bus never gets disconnected. It waits for the bus to make room, which the time limits of the clients that the bus waits on bound. * The bus sends its NameOwnerChanged, NameAcquired and NameLost signals on its own connection, as it sends its replies and its other signals, rather than straight to their recipients. So the one task that forwards all the messages of the bus forwards those too, and every recipient gets the messages of the bus in the order that the bus sent them. The peers hold on to that connection until the bus goes, as the task that serves the bus's own peer, which holds the peers, only ends once that connection does. * GetConnectionCredentials no longer holds the lock of the peers while it waits for the connection of the peer, which a send holds. This trades a limit on how many bytes may wait for a client, the 127 MiB of the queue that each client had, for a limit on how long a client may hold up those who send to it. dbus-broker limits bytes, with quotas for each user, and disconnects a client that can't take a reply or a signal that it expects, as the bus now does once it has waited on the client for the time limit. dbus-broker rejects a method call that the callee can't take, though, with LimitsExceeded, and keeps the callee, which the bus can't do: a send that times out may have written part of the call to the callee's socket. The time limit is long enough for a client that reads to take the largest message, of 128 MiB, at 128 MiB/s, and short next to the 25 s that a method call waits for its reply by default with libdbus, GDBus and sd-bus, as a client that stops reading while it is subscribed to the signals of the bus holds up every message of the bus for that long before it gets disconnected, and for as many times that long on a signal for several such clients. The cost is that the bus reads the messages of a client only as fast as their recipients take them, beyond the 64 messages that zbus reads ahead on each connection. A client that stops reading while it waits for the bus to take its messages, as clients on zbus 4.3 do, can then get disconnected for waiting on itself: when it has about 100 calls under way at once, on one connection, to an object that the same connection serves, the bus waits for it to take one of those calls while it waits for the bus to take the next ones, until the time limit is up. The queue took those calls, as dbus-broker does, within its quotas. And a client that stops reading for longer than the time limit, as one stopped in a debugger does, gets disconnected as soon as a message for it waits that long, where dbus-broker queues the messages for it up to its quota. Bus::set_send_timeout sets the limit for the clients that connect from then on, which the tests use to keep their limits short. tests/stalled_client.rs gets back the tests of the queue, adapted to the time limit: a client that stops reading holds up its senders until the limit and gets disconnected then, while the clients that read get every signal, in order, and the bus answers everyone, and a client that the bus can't write to, or that can't take file descriptors, is handled as before. And there are new tests of a client that reads again before the limit, of the names of a client that the bus disconnects, of a monitor that stops reading, of several sends timing out at once, of a client that goes while the bus waits on it, of the order of the messages of clients and of the bus, of other clients going on while a sender waits, and, on Linux, of a client that the bus wrote part of a message to when it timed out, which must get nothing after that part. tests/shutdown.rs tests that nothing is left of a bus once it goes. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
The task that serves a client forwarded each of its messages to the recipients of the message one after the other, waiting for each to take it, for up to the time limit of that recipient. So clients that don't read, which the same message went to, held it up for the time limit of each of them in turn: with 3 of them subscribed to the signals of the bus, a message of the bus waited 3 s before the bus went on. The task now sends a message to all of its recipients at once, and waits for all of them, so those that don't read hold it up for as long as the slowest of them: one time limit, if they stop taking messages at the same one. It still goes on to the next message only once each recipient has taken this one, or been disconnected, so each recipient still gets the messages of each sender in the order that the sender sent them, and the bus still queues nothing for anyone. A new test has 5 clients that stop reading take a flood of signals, and checks that the bus disconnects them all within less than 4 of their time limits. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
A client that stops reading holds up those who send to it for the time limit of the bus, before the bus disconnects it, and one that is subscribed to the signals of the bus holds up every message of the bus, its replies included. With a limit of 1 s, that holdup was long enough for people to notice. At 100 ms, it is barely noticeable. The cost is that a client has to keep up with what it gets: one that reads at 128 MiB/s still takes messages of up to 12.8 MiB within the limit, but the largest messages, of 128 MiB, now take reading at 1.25 GiB/s. And a client that gets stuck for longer than 100 ms, with a message waiting for it that its socket can't take, gets disconnected, where dbus-broker would queue the messages for it, up to its quota. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
busd puts a time limit on each send to a client through zruntime's Runtime::timeout, which read the clock as it made each time limit. Where reading the clock is a system call, as with the HPET clocksource, that cost busd 13 to 27% more CPU time than the queue that the time limit replaces. z-galaxy/zruntime#58 starts the clock only once a send has to wait, and no release has it yet, so build on its commit until one does. The commit takes the place of the release of zruntime for zbus too, through a patch of crates.io's zruntime, as busd and zbus have to share one runtime. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
This was referenced Oct 24, 2026
Merged
zeenix
added a commit
to zeenix/dbus-bench
that referenced
this pull request
Oct 24, 2026
The branch of zeenix/busd that had the commits that the comparison of busd's time limit measures was rebased onto busd's main, for z-galaxy/busd#382, so fetching the branch no longer brings them, and scripts/compare-send-timeout.sh stopped on a fresh checkout of busd. It now fetches each commit by its full hash, which GitHub still gives, and falls back to the branch only for a short hash, which a fetch cannot name. A short hash of a commit that the branch no longer has fails with a message that says to name the commit in full. A run with ZRUNTIME_REV no longer needs the revert, which it does not build. And as the head of busd's branch now patches zruntime itself, for a commit that no release has, the patch of a run with ZRUNTIME_REV goes in the Cargo configuration of the export, which takes precedence over the manifest, rather than in a second patch of zruntime in the manifest, which Cargo rejects as a duplicate key. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A client that stopped reading used to freeze the bus. #376 fixed that with a queue of up to 127 MiB for each client. This replaces the queue with a limit on how long a client may hold up those who send to it:
SEND_TIMEOUT), rather than 1 s.Measurements
On a Ryzen 7 7840U, against busd's main (the queue), dbus-broker and dbus-daemon (results):
sendmmsg(), which zbus has no API for yet.With clients that stop reading, frozen subscribers of the bus's signals hold up the bus for about 100 ms, however many of them there are (results on a VM).
Costs
Generated by Claude Opus 5.5.