Skip to content

🐛 Queue messages for each client rather than wait on it - #376

Merged
zeenix merged 2 commits into
z-galaxy:mainfrom
zeenix:fix-stalled-reader
Oct 21, 2026
Merged

zeenix merged 2 commits into
z-galaxy:mainfrom
zeenix:fix-stalled-reader

Conversation

@zeenix

@zeenix zeenix commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

A client that stops reading its socket froze the whole bus. Each connection has one task that reads a message from its client and forwards it before reading the next, and a send only returns once the destination's socket has taken the whole message. So such a client held up everyone who sent to it, the bus itself included. Once a broadcast of the bus waited on it, holding the read locks of the name registry and the peers, the next client to connect waited for the write lock of the peers. zruntime's RwLock prefers writers, so every forward after that waited too.

With a client subscribed to the bus's signals stopped by SIGSTOP (gdbus monitor under kill -STOP), the bus stopped answering anyone after 66 clients had come and gone, calls between clients that were connected all along included, until the frozen client was let go. dbus-daemon and dbus-broker answered all 1500 calls of the same test. The same mechanism deadlocked a benchmark where 100 proxies on one zbus 4.3 connection call a service on that same connection.

The change

  • Each client gets a queue of the messages on their way to it, and a task that writes the queue to its socket. Forwarding to a client only queues the message, so it never waits on the client's socket. Locks are held only while queuing.
  • Messages for the bus itself still go straight to its connection and may wait for room, which holds back a client that floods the bus. That is safe because the bus never waits on a client any more, and such a message is only sent once the locks are released, since the bus may need them to make room.
  • A queue always takes one message, however big, but a client whose queue would hold more than 127 MiB (dbus-daemon's default max_outgoing_bytes) gets disconnected, so that a client that never reads can't make the bus hold its messages for good. The constant could come from the config's <limit> later; busd ignores those for now.
  • A message with file descriptors, for a client that didn't negotiate passing them, is dropped, as on main, and the messages after it still go. Any other failure to write to a client gets it disconnected, so that the bus doesn't keep a client that gets nothing, along with its names.
  • Passing file descriptors is part of the write, and Linux passes at most 253 (SCM_MAX_FD) in one write, while zbus passes all those of a message with its first write. So passing on a message with more always fails, and would get its recipient disconnected: any client could get any other one disconnected. The second commit disconnects the client that sends such a message instead, as dbus-broker 35 does. dbus-daemon 1.14.10, with its stock session configuration, disconnects the sender of a message with 17 or more.
  • zruntime's mpmc feature provides the queue.
  • A known limit: a client that shuts its socket down for writing while a message is on its way to it, and never reads, leaves that message's writer waiting, and with it the client's socket and queue, until busd exits. Before this change, the task of whoever sent the message got stuck the same way, holding up its client too. Having the writer stop when the last handle of its queue is dropped would fix it.

Testing

  • tests/stalled_client.rs adds five tests:
    • stalled_client: a client that reads nothing while subscribed to every signal gets 2 MiB of signals, and the bus must still answer RequestName/ReleaseName and a new client.
    • overflowing_client: signals of 8 MiB go to such a client until it gets disconnected.
    • client_without_fds: a client on a socket that zbus doesn't negotiate passing file descriptors over gets a signal with one, and must still get the reply to its next call.
    • unwritable_client, on Linux only, where writing to a socket fails once its peer shuts it down for reading: a client does that, gets a signal, and must get disconnected.
    • too_many_fds, on Linux only: a client sends another one a message with 253 file descriptors, which must get to it, and then one with 254, for which the bus must disconnect the sender, and not the recipient.
  • The first two time out against main, the next two fail if the writer stops at a failed write without disconnecting the client, and the last one fails without the second commit. All five pass, within about a second, in each of 320 runs, with other runs going at the same time.
  • cargo test on stable and nightly, cargo +nightly fmt --all -- --check, cargo clippy --all -- -D warnings, with and without default features and with --all-targets, cargo check --tests --target aarch64-apple-darwin, the builds of CI with RUSTFLAGS=-D warnings and the gimoji commitlint pass, on each commit. The branch merges cleanly with zeenix/busd fix-monitor-broadcast and match-rule-fixes, and all tests pass with the three merged.
  • The frozen-subscriber test (in Refresh the results on zbus 6, and add a look into a busd stall zeenix/dbus-bench#3) now gets the same results as dbus-daemon and dbus-broker: all 1500 calls answered, and 8 of 8 calls between long-lived clients.
  • The benchmark client on zbus 4.3 that deadlocked in its shared-connection phase now finishes it, in all 3 runs (11.1 to 11.8 s).

Cost

60 interleaved rounds of the dbus-bench benchmark (zbus 6 client) on a 4-vCPU VM, against main (a982812), with Mann–Whitney U p-values:

Metric main this branch
Single client 0.139 s 0.139 s (p = 0.65)
20 clients 0.684 s 0.694 s (p = 0.37)
Broker CPU time, 20 clients 0.480 s 0.490 s (+2.1%, p = 0.0008)
Peak memory 4826 KiB 4896 KiB (+1.5%, p < 0.0001)

Generated by Claude Opus 5.5 (claude-opus-5-5).

🤖 Generated with Claude Code

@zeenix zeenix left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested.

Review by Codex on behalf of @zeenix.

One blocking regression: an unsupported file-descriptor message permanently disables delivery to an otherwise live client.

Comment thread src/peer/outgoing.rs Outdated
Each connection had one task, which read a message from its client and
sent it on before reading the next, and a send only returned once the
destination's socket had taken the whole message. So a client that
stopped reading its socket held up every client that sent to it, the
bus itself included. And once a broadcast of the bus waited on such a
client, holding the read locks of the name registry and of the peers,
the next client to connect waited for the write lock of the peers, and
as the lock prefers writers, every message forwarded after that waited
behind it. With a client that is subscribed to the bus's signals
stopped by SIGSTOP, the bus stopped answering anyone after 66 clients
had come and gone.

Give each client a queue of the messages on their way to it, and a task
that writes the queue to its socket, so that forwarding a message to a
client never waits, and only queue messages while holding the locks. A
message for the bus itself still waits for room on its connection, as
the bus no longer waits on any client, but only once the locks are
released, as the bus may need them to make room.

A queue always takes one message, but a client whose queue would hold
more than 127 MiB, as many bytes as dbus-daemon's default
max_outgoing_bytes, gets disconnected, so that a client that never
reads cannot have the bus hold on to its messages for good.

A message with file descriptors, for a client that didn't negotiate
passing them, is dropped, as before. A client also gets disconnected
when writing to its socket fails otherwise, so that the bus doesn't
keep a client that gets nothing, along with its names.

Add tests where a client stops reading: they hang without this change.
Also test that a message with file descriptors doesn't hold up the
messages after it, and, on Linux, that a client whose socket can't be
written to gets disconnected.

On a 4-vCPU VM, the benchmark of zeenix/dbus-bench measures the same
timings with this change, with 2.1% more CPU time in its 20-client
phase and 70 KiB more peak memory.

Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
@zeenix
zeenix force-pushed the fix-stalled-reader branch from 5a867b2 to f8923bb Compare October 21, 2026 21:08
The bus disconnects a client when writing to its socket fails, and
passing the file descriptors of a message is part of writing it. Linux
passes at most 253 file descriptors (SCM_MAX_FD) in one write, and zbus
passes all those of a message with its first write, so passing on a
message with more always fails. So any client could get any other one
disconnected, by sending it a message with more file descriptors, split
over several writes.

Disconnect the client that sends such a message instead, as dbus-broker
35 does. dbus-daemon 1.14.10, with its stock session configuration,
disconnects the sender of a message with 17 or more. A message with 253
still goes through.

Add a test, on Linux only, where a client sends another one a message
with 253 file descriptors, which gets to it, and then one with 254,
which gets the sender disconnected and not the recipient. It fails
without this change.

Assisted-by: Claude Opus 5.5 (claude-opus-5-5)

@zeenix zeenix left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No actionable findings.

Review by Codex on behalf of @zeenix.

@zeenix
zeenix merged commit a4c12aa into z-galaxy:main Oct 21, 2026
7 checks passed
@zeenix
zeenix deleted the fix-stalled-reader branch October 21, 2026 22:17
zeenix added a commit to zeenix/dbus-bench that referenced this pull request Oct 22, 2026
A branch of busd reverts the queue for each client of
z-galaxy/busd#376 and gives each send to a client a time limit instead.
This compares it with busd's main branch, with the revert alone and
with dbus-broker: in pairs, on the benchmark, on signals, on latencies
and on a connection that 100 proxies share, on zbus 6, and on clients
that stop reading, as frozen subscribers of the bus's signals and as
the shared connection of the zbus 4.3 client. Along with an experiment
with reading more messages ahead from each client.

Assisted-by: Claude Opus 5.5 (claude-opus-5-5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant