Skip to content

Keep RabbitMQ retries local and honor dead-lettering on exhaustion - #105

Draft
niemyjski wants to merge 7 commits into
feature/rabbitmq-broker-verificationfrom
feature/rabbitmq-delivery-recovery
Draft

niemyjski wants to merge 7 commits into
feature/rabbitmq-broker-verificationfrom
feature/rabbitmq-delivery-recovery

Conversation

@niemyjski

@niemyjski niemyjski commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Classic retries now return only to their source queue and are confirmed before acknowledgement. Recovery cancels stale Automatic handlers, respects subscription overrides, reports retained retries accurately, and reuses healthy connections after broker rejection.

Behavior changes: Automatic handlers now receive cancellation when their delivery channel is lost. Cancellation is cooperative and cannot undo side effects; handlers must tolerate duplicates. FireAndForget remains the default, and its already-acknowledged handlers may finish after channel loss. Ordinary Automatic exhaustion uses broker dead-lettering or discard; classic exhaustion now honors configured DLX routing instead of acknowledging and discarding. RequireSuccessfulDispatch opts into confirmed terminal transfers that retain work while the destination is unavailable.

Compatibility: Classic retries require write permission on amq.default, which may require an ACL change. Invalid configurations fail earlier, and shutdown waits are bounded. Local retry counts do not survive connection loss; unlimited retries can block later work. Retained local retries now report not-ready and expose their delivery error. The subscription-override compatibility regression is fixed. This correction adds no new public options or queue-migration requirement.

Validation: Release builds for .NET 8/10 and formatting passed. Local verification passed 61 lifecycle/handoff cases, including 25 new cases covering classic/quorum on RabbitMQ 4.2.5/4.3.6, with zero failures or skips. Full repository CI passed at 754c43e: 429 passed, zero failed, five existing priority-test skips.

Verification and implementation details

The same 61 lifecycle/handoff cases also pass against Foundatio 13.0.5 (current main) in an isolated compatibility runner, with zero skips. No dependency versions were changed in this PR.

Five new test methods cover channel-loss cancellation, unchanged FireAndForget handling, subscription setup/maintenance overrides, retained-retry diagnostics, and transport reuse after NACKs or mandatory returns. Existing test methods are unchanged by this correction. The four defects first produced 17 failing cases, with four strict-dispatch controls passing; the added compatibility guards also caught and prevented cancellation of already-acknowledged FireAndForget work.

The reuse regression compares socket counts after the first rejection, so IPv6-to-IPv4 fallback does not produce a false failure. All eight reuse cases also pass using localhost.

Local verification runs the exact repository test bodies against disposable brokers. It excludes the two management-API-dependent methods; the full repository CI uses the owned Aspire fixture, including permissions, TLS, delayed delivery, process probes, and recovery tests. The only local build warning is the existing ASPIRE010. No rolling-upgrade, throughput, or allocation-neutrality claim is made.

The ordinary shared message dispatcher is retained by this correction. Scoped async context carries the originating delivery token through its scheduled callbacks and the caller's subscription context to the existing virtual setup hook. Maintenance also honors that hook and its documented disposal-token contract. These fixes introduce no dependency upgrade, additional dispatcher, partial class, or test infrastructure project. Normal publication rejection preserves an open handoff channel and connection; transport failures still reconnect.

Classic retry handoff uses the default exchange and actual source queue, preserving logical identity. A broker NACK retries locally while retaining the original and consuming the same finite delivery budget, allowing a full source queue to drain after success or exhaustion. Unroutable returns, permission failures, and ambiguous confirmations retain the original until repair or cancellation. Handoff confirmation and source acknowledgement are not atomic, so replay can duplicate delivery.

Ordinary exhaustion uses BasicReject(requeue: false), honoring queue arguments or broker policies and producing native x-death metadata. Broker dead-lettering is at-most-once by default; quorum queues can opt into broker-managed at-least-once transfer. Neither mode auto-creates an error queue. Required dispatch needs Automatic acknowledgements and an explicit DeadLetterExchange; its confirmed application transfer uses provider failure headers. See RabbitMQ dead-letter safety.

Earlier validation changes reject retry limits below -1, invalid recovery/heartbeat intervals, and a terminal exchange equal to the source during construction. Changing an explicit queue type after construction fails before declaration. Body copies preserve ownership when handlers outlive callbacks, and public header constants retain their wire names. Native delayed retries and per-queue consumer timeout retain their 4.3+ restrictions; older-version coverage remains in place.

Default-exchange write permission broadens publishing access within the vhost, so use an appropriately isolated vhost. Production code requires no management permission. See RabbitMQ access control and connection reuse guidance.

Merge after #104 and before #100.

@niemyjski
niemyjski added this pull request to stack #107 September 25, 2026 20:21
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch 2 times, most recently from 6923527 to 962b7f7 Compare September 25, 2026 20:39
@niemyjski niemyjski changed the title Prevent RabbitMQ delivery loss during retries and recovery Retain exhausted RabbitMQ deliveries and prevent retry loss Sep 25, 2026
@niemyjski niemyjski changed the title Retain exhausted RabbitMQ deliveries and prevent retry loss Prevent Automatic retry loss and retain exhausted deliveries Sep 25, 2026
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from 962b7f7 to f34aedc Compare September 25, 2026 21:29
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from f34aedc to b7c1a1f Compare September 25, 2026 21:43
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from b7c1a1f to 2d7f928 Compare September 25, 2026 21:49
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch 2 times, most recently from 2995cf3 to d6aca4a Compare September 26, 2026 02:20
@niemyjski niemyjski changed the title Prevent Automatic retry loss and retain exhausted deliveries Retain exhausted RabbitMQ deliveries and confirm retry handoffs Sep 26, 2026
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch 2 times, most recently from 6bcdb4d to c170bb3 Compare September 29, 2026 02:04
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from c170bb3 to 1528a24 Compare October 2, 2026 03:28
@niemyjski niemyjski changed the title Retain exhausted RabbitMQ deliveries and confirm retry handoffs Keep RabbitMQ retries local and honor broker dead-lettering Oct 5, 2026
@niemyjski niemyjski changed the title Keep RabbitMQ retries local and honor broker dead-lettering Keep RabbitMQ retries local and prevent full-queue stalls Oct 6, 2026
@niemyjski niemyjski changed the title Keep RabbitMQ retries local and prevent full-queue stalls Keep RabbitMQ retries local and honor dead-lettering on exhaustion Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant