Repository navigation
🐛 Queue messages for each client rather than wait on it - #376
Merged
Merged
Conversation
This was referenced Oct 21, 2026
Each connection had one task, which read a message from its client and sent it on before reading the next, and a send only returned once the destination's socket had taken the whole message. So a client that stopped reading its socket held up every client that sent to it, the bus itself included. And once a broadcast of the bus waited on such a client, holding the read locks of the name registry and of the peers, the next client to connect waited for the write lock of the peers, and as the lock prefers writers, every message forwarded after that waited behind it. With a client that is subscribed to the bus's signals stopped by SIGSTOP, the bus stopped answering anyone after 66 clients had come and gone. Give each client a queue of the messages on their way to it, and a task that writes the queue to its socket, so that forwarding a message to a client never waits, and only queue messages while holding the locks. A message for the bus itself still waits for room on its connection, as the bus no longer waits on any client, but only once the locks are released, as the bus may need them to make room. A queue always takes one message, but a client whose queue would hold more than 127 MiB, as many bytes as dbus-daemon's default max_outgoing_bytes, gets disconnected, so that a client that never reads cannot have the bus hold on to its messages for good. A message with file descriptors, for a client that didn't negotiate passing them, is dropped, as before. A client also gets disconnected when writing to its socket fails otherwise, so that the bus doesn't keep a client that gets nothing, along with its names. Add tests where a client stops reading: they hang without this change. Also test that a message with file descriptors doesn't hold up the messages after it, and, on Linux, that a client whose socket can't be written to gets disconnected. On a 4-vCPU VM, the benchmark of zeenix/dbus-bench measures the same timings with this change, with 2.1% more CPU time in its 20-client phase and 70 KiB more peak memory. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
zeenix
force-pushed
the
fix-stalled-reader
branch
from
October 21, 2026 21:08
5a867b2 to
f8923bb
Compare
The bus disconnects a client when writing to its socket fails, and passing the file descriptors of a message is part of writing it. Linux passes at most 253 file descriptors (SCM_MAX_FD) in one write, and zbus passes all those of a message with its first write, so passing on a message with more always fails. So any client could get any other one disconnected, by sending it a message with more file descriptors, split over several writes. Disconnect the client that sends such a message instead, as dbus-broker 35 does. dbus-daemon 1.14.10, with its stock session configuration, disconnects the sender of a message with 17 or more. A message with 253 still goes through. Add a test, on Linux only, where a client sends another one a message with 253 file descriptors, which gets to it, and then one with 254, which gets the sender disconnected and not the recipient. It fails without this change. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
zeenix
added a commit
to zeenix/dbus-bench
that referenced
this pull request
Oct 22, 2026
A branch of busd reverts the queue for each client of z-galaxy/busd#376 and gives each send to a client a time limit instead. This compares it with busd's main branch, with the revert alone and with dbus-broker: in pairs, on the benchmark, on signals, on latencies and on a connection that 100 proxies share, on zbus 6, and on clients that stop reading, as frozen subscribers of the bus's signals and as the shared connection of the zbus 4.3 client. Along with an experiment with reading more messages ahead from each client. Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A client that stops reading its socket froze the whole bus. Each connection has one task that reads a message from its client and forwards it before reading the next, and a send only returns once the destination's socket has taken the whole message. So such a client held up everyone who sent to it, the bus itself included. Once a broadcast of the bus waited on it, holding the read locks of the name registry and the peers, the next client to connect waited for the write lock of the peers. zruntime's
RwLockprefers writers, so every forward after that waited too.With a client subscribed to the bus's signals stopped by SIGSTOP (
gdbus monitorunderkill -STOP), the bus stopped answering anyone after 66 clients had come and gone, calls between clients that were connected all along included, until the frozen client was let go. dbus-daemon and dbus-broker answered all 1500 calls of the same test. The same mechanism deadlocked a benchmark where 100 proxies on one zbus 4.3 connection call a service on that same connection.The change
max_outgoing_bytes) gets disconnected, so that a client that never reads can't make the bus hold its messages for good. The constant could come from the config's<limit>later; busd ignores those for now.main, and the messages after it still go. Any other failure to write to a client gets it disconnected, so that the bus doesn't keep a client that gets nothing, along with its names.SCM_MAX_FD) in one write, while zbus passes all those of a message with its first write. So passing on a message with more always fails, and would get its recipient disconnected: any client could get any other one disconnected. The second commit disconnects the client that sends such a message instead, as dbus-broker 35 does. dbus-daemon 1.14.10, with its stock session configuration, disconnects the sender of a message with 17 or more.mpmcfeature provides the queue.Testing
tests/stalled_client.rsadds five tests:stalled_client: a client that reads nothing while subscribed to every signal gets 2 MiB of signals, and the bus must still answerRequestName/ReleaseNameand a new client.overflowing_client: signals of 8 MiB go to such a client until it gets disconnected.client_without_fds: a client on a socket that zbus doesn't negotiate passing file descriptors over gets a signal with one, and must still get the reply to its next call.unwritable_client, on Linux only, where writing to a socket fails once its peer shuts it down for reading: a client does that, gets a signal, and must get disconnected.too_many_fds, on Linux only: a client sends another one a message with 253 file descriptors, which must get to it, and then one with 254, for which the bus must disconnect the sender, and not the recipient.main, the next two fail if the writer stops at a failed write without disconnecting the client, and the last one fails without the second commit. All five pass, within about a second, in each of 320 runs, with other runs going at the same time.cargo teston stable and nightly,cargo +nightly fmt --all -- --check,cargo clippy --all -- -D warnings, with and without default features and with--all-targets,cargo check --tests --target aarch64-apple-darwin, the builds of CI withRUSTFLAGS=-D warningsand the gimoji commitlint pass, on each commit. The branch merges cleanly with zeenix/busdfix-monitor-broadcastandmatch-rule-fixes, and all tests pass with the three merged.Cost
60 interleaved rounds of the dbus-bench benchmark (zbus 6 client) on a 4-vCPU VM, against
main(a982812), with Mann–Whitney U p-values:Generated by Claude Opus 5.5 (claude-opus-5-5).
🤖 Generated with Claude Code