Resilience layer by swnsma · Pull Request #245 · justbetter/magento2-sentry

swnsma · 2026-07-08T14:03:07Z

Summary
There is a major drawback in the PHP implementation: communication with Sentry interferes with shoppers' critical paths.
Consequently, if Sentry goes down (that is a case especially with self-hosted Sentry), the whole website will experience the issue.
We experienced the issue on production, where the current implementation caused 5-15s of extra loading time on all storefront pages.

Related issue
#246

Result
This pull request adds several features to improve the resilience of the extension:

async message processing (via Magento's MQs) - a new topology with its own consumer
Circuit breaker on async paths - temporarily shut down communication with Sentry (or redirect it into the async path) if the service becomes unavailable
http/connection timeout settings

Checklist

I've ran composer run codestyle
I've ran composer run analyse

- async communication with sentry - cirquite breaker if Sentry is out - cover changes with tests

Enforced strict sync or async delivery modes in ResilientTransport. Dropped events immediately when circuit was open or sync HTTP failed instead of falling back to message queue. Cleaned up documentation slop.

…on from module.xml

Resilience layer

amenk · 2026-08-06T14:49:54Z

I like the idea.

The MR also contains a AGENTS.md - should maybe be separated and discussed in another PR.

indykoning

Thanks for the meaningful contribution!
While a broken sentry server does not break the frontend it does slow it down.
As i do want to thoroughly test it i won't be able to merge it yet, but i hope to get to it soon 🙂

…ayer Keep resilient transport and wire upstream TrackingHttpClient through it so the admin test button can still inspect the last Sentry HTTP response.

indykoning

I've tested it out, in my case i'm observing 4 messages when i run into a frontend error:

The exception
The Exception log message
Profiling information
Log messages

It's a lot of data for a single request, 4 messages with a large payload.
However this is not something we can/should improve on, i think we may want to warn the user to set good message lifetimes and ensure the message queue cleanup is running when using MysqlMQ.

It looks really promising from my testing!

indykoning · 2026-08-19T15:13:34Z

+    <consumer name="justbetter.sentry.event"
+              queue="justbetter.sentry.event"
+              handler="JustBetter\Sentry\Model\Queue\Consumer\SentryEventConsumer::process"
+              consumerInstance="Magento\Framework\MessageQueue\Consumer"
+              maxMessages="500"/>


Looking at Magento's queue_consumer.xml files i don't see the consumerInstance or maxMessages
I think i'd prefer that as well so it's up to the user what value they want here.

Suggested change

<consumer name="justbetter.sentry.event"

queue="justbetter.sentry.event"

handler="JustBetter\Sentry\Model\Queue\Consumer\SentryEventConsumer::process"

consumerInstance="Magento\Framework\MessageQueue\Consumer"

maxMessages="500"/>

<consumer name="justbetter.sentry.event"

queue="justbetter.sentry.event"

handler="JustBetter\Sentry\Model\Queue\Consumer\SentryEventConsumer::process"/>

indykoning · 2026-08-19T15:17:55Z

+        // Never wrap our own async Sentry delivery — that would nest envelopes and risk recursion.
+        if ($topic === SentryEventPublisher::TOPIC_NAME) {
+            return [$topic, $envelopes];
+        }


Do we need this for more listeners?
Cases i can think of (but luckily haven't observed) are:

Errors thrown while running the queue message

Log messages written while running the queue message

Good point.
I have added a sending guard to prevent reporting logs from being written while scheduling a message or in the consumer.

indykoning · 2026-08-19T15:24:31Z

+
+Event delivery can be configured as either synchronous or asynchronous:
+
+1. (Default) **Sync mode + circuit breaker** (`async_sending_enabled = false`): Events are sent over HTTP immediately. If `circuit_breaker_enabled` is true and consecutive failures reach `circuit_breaker_failure_threshold`, the circuit opens and subsequent requests fail fast without blocking Magento.


One of my preferences would've been to also unblock the user by deferring the sending to after the response has been sent.

Which works fantastic with Symfony & Laravel due to the fastcgi_finish_request():
https://github.com/symfony/http-foundation/blob/a3cef50a1c6c37b60e0457d2152c2c572dc901cb/Response.php#L408

Unfortunately this implementation isn't what Magento uses:
https://github.com/laminas/laminas-http/blob/9462fc84330d25b23383823831380abb33907fdd/src/PhpEnvironment/Response.php#L147

So i think this is indeed the best we can do.

I also looked into it, but I was concerned that it comes as part of php-fpm, so it won't help in an non-fpm environment

indykoning · 2026-08-19T15:37:40Z

      "magento/composer-dependency-version-audit-plugin": true,
-      "dealerdirect/phpcodesniffer-composer-installer": true
+      "dealerdirect/phpcodesniffer-composer-installer": true,
+      "magento/magento-composer-installer": true


I think the composer plugin here is causing the checks to fail, do we need it or can we safely disable it in the package?

- remove consumer configs from queue_consumer.xml - remove plugin-allow from composer.json - introduce a delivery guard to don't send messages while messages sending is in progress.

- Consumer no longer throws; delivery failures logged to var/log/sentry.log channel excluded from Sentry by MonologPlugin. - CircuitBreaker opens immediately on 429 until Sentry retry time (new RateLimitParser); cache TTL covers backoff. - EnvelopeSender feeds breaker: 2xx success, 429 rate limit, 5xx/cURL failure, other 4xx ignored. - Consumer and async publish skip sending while circuit open. - Removed circuit_breaker_enabled config and Data::isCircuitBreakerEnabled(); breaker always on.

The consumer no longer throws and logs delivery failures to a channel that is excluded from Sentry, so it can't feed its own failures back into the queue. Any remaining capture during delivery is bounded by the circuit breaker, which opens on Sentry's 429 and stops async publishing. A process-wide guard only hid real errors raised while a message was being delivered.

A 429 for one category, e.g. an exhausted transaction quota, opened the circuit breaker for everything, so errors and cron check-ins were dropped as well and Sentry Crons reported missed check-ins. Rate limits now live in a cache-backed RateLimitState keyed by category, the same way the SDK's in-memory RateLimiter keys them, and the circuit breaker only tracks whether Sentry is reachable. Both delivery modes record responses through ResponseRecordingHttpClient, so sync mode now also honours X-Sentry-Rate-Limits / Retry-After across requests.

novikor · 2026-10-05T16:45:35Z

We pushed a follow-up round to the resilience layer. Here is what the five new commits do and why.

1. env.php values were ignored for the resilience settings (89b89dc)
Helper\Data reads sentry/environment/* from the store config, and only falls back to env.php when that value is null. The new resilience fields had non-empty defaults in etc/config.xml, so anything set in env.php was silently ignored. Their tags are now empty, like the other environment fields. The defaults live in Helper\Data::$configKeys and are documented in the README table.

2. Fixed a 429 feedback loop in async mode (7e28b57)
When Sentry answered 429 because the quota was exhausted, the consumer threw. Magento then rejected the message and logged the rejection through a logger with the Sentry handler attached, which queued yet another event. Now:

the consumer never throws. Delivery failures go to var/log/sentry.log on a channel that MonologPlugin skips;
the circuit breaker is always on, and the circuit_breaker_enabled setting is removed.

3. Rate-limit responses are no longer logged (ce0c351)
They are expected while the quota is exhausted, and delivery already backs off.

4. Replaced DeliveryGuard with a transport-local re-entrancy flag (3006c39)
@indykoning, this follows up on your question in the ExchangePlugin.php thread about errors and log messages written while a queue message is processed. With commit 2, the cases you listed can no longer loop:

the consumer no longer throws;
its own delivery failures are logged to a channel that is excluded from Sentry;
queue tracing already skips the module's own topic.

Anything else captured during delivery is now bounded by the circuit breaker and the rate limits from commit 5. A process-wide guard would mostly have hidden real errors raised while a message was delivered.

5. Sentry rate limits are tracked per data category (5d78977)
A 429 for one category, for example an exhausted transaction quota, used to stop everything. Errors and cron check-ins were dropped too, and Sentry Crons reported missed check-ins. The SDK's own RateLimiter keys limits by category, but it only keeps them in memory for one request, and the async consumer bypasses it. Now:

RateLimitState keeps the X-Sentry-Rate-Limits / Retry-After limits per category in the Magento cache, so every process sees them. A limit on one category doesn't block the others;
CircuitBreaker only tracks whether Sentry is reachable (5xx and network errors);
both delivery modes record every response through ResponseRecordingHttpClient. As a result, sync mode also respects rate limits across requests, which it didn't before.

The README section "Resilient delivery" is updated accordingly. All unit tests in Test/Unit pass. That includes a test that serializes real SDK envelopes for each category and checks that only the limited category is held back.

Known issue: the admin "Test" button in async mode
The delivery check added in #249 reads the last HTTP response of the current request. In async mode the event is only queued, so there is no response, and the button reports "Sentry did not confirm delivery" even when everything works. If you're fine with it, we'll fix this in a separate PR once this one is handled, to keep this one from growing further.

novikor · 2026-10-06T07:30:06Z

Maybe could move rate limits handling into a separate PR, btw

swnsma · 2026-10-06T11:31:40Z

Hi @novikor,
Thank you very much for syncing the PR with the latest changes and fixes!

@indykoning, could you have a look and let us know if there is anything else that should be addressed in this PR?
We would also appreciate if you could share an approximate time expectations for when the PR might be merged. Once it is merged, we can stop maintaining our fork of the repo and start contributing futher bugfixes/improvements directly to the upstream.

Thank you!

Introduce resilience layer:

97b6dbb

- async communication with sentry - cirquite breaker if Sentry is out - cover changes with tests

swnsma force-pushed the resilience-layer branch from 26272c0 to 97b6dbb Compare July 9, 2026 18:59

swnsma mentioned this pull request Jul 10, 2026

[⭐]: The extension blocks critical path #246

Open

2 tasks done

novikor and others added 3 commits July 20, 2026 13:22

[IE-372] refactor: removed queue fallback for 2-state delivery

e6a3fcb

Enforced strict sync or async delivery modes in ResilientTransport. Dropped events immediately when circuit was open or sync HTTP failed instead of falling back to message queue. Cleaned up documentation slop.

[IE-372] Updated readme, fixed system.xml and removed redundant versi…

eb9c2c5

…on from module.xml

Merge pull request #1 from novikor/resilience-layer

753f183

Resilience layer

indykoning reviewed Aug 12, 2026

View reviewed changes

Comment thread AGENTS.md Outdated

swnsma force-pushed the resilience-layer branch from b15b8db to 753f183 Compare August 14, 2026 13:49

swnsma added 2 commits August 14, 2026 16:52

Merge branch 'master' of justbetter/magento2-sentry into resilience-l…

3298e1f

…ayer Keep resilient transport and wire upstream TrackingHttpClient through it so the admin test button can still inspect the last Sentry HTTP response.

Resolve merge conflict.

8659599

indykoning reviewed Aug 19, 2026

View reviewed changes

swnsma force-pushed the resilience-layer branch from d4ae459 to 79c7efb Compare August 25, 2026 14:46

justbetter#246.

806cdd4

- remove consumer configs from queue_consumer.xml - remove plugin-allow from composer.json - introduce a delivery guard to don't send messages while messages sending is in progress.

swnsma force-pushed the resilience-layer branch from 79c7efb to 806cdd4 Compare August 25, 2026 14:48

novikor added 5 commits October 5, 2026 17:35

Fixed env.php values ignored over config.xml defaults

89b89dc

Skipped logging of Sentry rate-limit responses

ce0c351

Applied StyleCI fixes

0bf89d2

Navigation Menu

Repository navigation

Resilience layer - #245

Resilience layer#245
swnsma wants to merge 13 commits into
justbetter:masterfrom
swnsma:resilience-layer

swnsma commented Jul 8, 2026 •

edited

Loading

Uh oh!

amenk commented Aug 6, 2026

Uh oh!

indykoning left a comment

Uh oh!

Uh oh!

indykoning left a comment

Uh oh!

indykoning Aug 19, 2026

Uh oh!

indykoning Aug 19, 2026

Uh oh!

swnsma Aug 25, 2026

Uh oh!

indykoning Aug 19, 2026

Uh oh!

swnsma Aug 25, 2026

Uh oh!

indykoning Aug 19, 2026

Uh oh!

novikor commented Oct 5, 2026

Uh oh!

novikor commented Oct 6, 2026

Uh oh!

swnsma commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		Event delivery can be configured as either synchronous or asynchronous:

		1. (Default) Sync mode + circuit breaker (`async_sending_enabled = false`): Events are sent over HTTP immediately. If `circuit_breaker_enabled` is true and consecutive failures reach `circuit_breaker_failure_threshold`, the circuit opens and subsequent requests fail fast without blocking Magento.

Conversation

swnsma commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amenk commented Aug 6, 2026

Uh oh!

indykoning left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

indykoning left a comment

Choose a reason for hiding this comment

Uh oh!

indykoning Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

indykoning Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

swnsma Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

indykoning Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

swnsma Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

indykoning Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

novikor commented Oct 5, 2026

Uh oh!

novikor commented Oct 6, 2026

Uh oh!

swnsma commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

swnsma commented Jul 8, 2026 •

edited

Loading