Skip to content

fix(repeater): stop reconnecting after the socket is torn down - #778

Merged
aborovsky merged 2 commits into
nextfrom
fix/repeater-reconnect-after-teardown
Sep 16, 2026
Merged

aborovsky merged 2 commits into
nextfrom
fix/repeater-reconnect-after-teardown

Conversation

@aborovsky

@aborovsky aborovsky commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

relates-to: #675

A reconnection timer could fire after `disconnect()` had already dropped
`_socket`, so `get socket` threw from inside `Timeout._onTimeout`. Nothing
can catch a throw there, so it surfaced as a fatal uncaught exception:
"Please make sure that repeater established a connection with host."

`handleConnectionError` emitted `ERROR` and then unconditionally scheduled a
reconnection. For `repeater_unauthorized` / `repeater_not_permitted` that emit
is dispatched synchronously into the consumer, which treats the code as
critical and calls `disconnect()` — so the reconnection was scheduled against
a socket that no longer existed. Restore the early return for those terminal
codes, since retrying an authorization rejection cannot succeed anyway.

`scheduleReconnection` also overwrote `connectionTimer` without clearing the
previous handle, leaking one timer per failed attempt. `disconnect()` could
only cancel the last one, so every leaked timer fired and threw. Clear before
scheduling to keep a single pending attempt.

Additionally:

- guard teardown with a `disposed` flag, checked before scheduling a
  reconnection and before the manual reconnect on `io server disconnect`
- reach for `_socket` instead of the throwing getter in deferred callbacks
  (reconnection timer, `deploy`'s `nextTick`), so a late callback is a no-op
- swap `MIN_RECONNECTION_DELAY` / `MAX_RECONNECTION_DELAY`, which were
  inverted (5s min vs 1s max). `Math.min(delay, MAX)` collapsed every backoff
  to 1s, and socket.io received `reconnectionDelay` above
  `reconnectionDelayMax`
- drop manager listeners on teardown so a later `connect()` cannot
  double-register them on the cached manager
@aborovsky

Copy link
Copy Markdown
Contributor Author

Summary

Fixes the fatal Sentry issue Error: Please make sure that repeater established a connection with host. (~76K events), which is thrown from get socket inside Timeout._onTimeout.

DefaultRepeaterServer has exactly one timer that dereferences the socket — the reconnection timer in scheduleReconnection:

this.connectionTimer = setTimeout(() => this.socket.connect(), delay);

this.socket throws whenever _socket is undefined, and _socket is only cleared by disconnect(). So the error means a reconnection timer fired after teardown. A throw inside a timer callback has no frame to catch it, hence Fatal / Unhandled.

How it happens

handleConnectionError emitted ERROR and then scheduled a reconnection unconditionally:

if (data && this.suppressConnectionError(data)) {
  this.events.emit(RepeaterServerEvents.ERROR, { ...data, message: err.message });
}

// Try reconnect in any case.
this.scheduleReconnection();

events.emit is synchronous and so is the whole downstream chain:

emit(ERROR) → wrapEventListener → ServerRepeaterLauncher.handleError → isCriticalError → handleCriticalError → close() → disconnect() → _socket = undefined

Control then returns to scheduleReconnection(), which installs a fresh timer on an already-disposed socket. It fires ~1s later and throws.

suppressConnectionError gates on REPEATER_UNAUTHORIZED / REPEATER_NOT_PERMITTED, and both are in the launcher's critical list — so every repeater started with a revoked token or without permission crashed this way, deterministically.

This was a regression from #675, which removed the return that used to end that branch.

Why the volume is so high

Two amplifiers, both from #675:

  1. scheduleReconnection overwrote connectionTimer without clearing the previous handle. Every failed attempt during an outage leaked another timer, and disconnect() could only cancel the last one — so all the leaked ones fired and threw.
  2. The delay bounds were inverted: MIN_RECONNECTION_DELAY = 5_000 vs MAX_RECONNECTION_DELAY = 1_000. Since the last step is Math.min(delay, MAX), the exponential backoff above it was dead code and every retry landed at 1s. socket.io also got reconnectionDelay: 5000 with reconnectionDelayMax: 1000.

Changes

  • Don't reconnect on terminal errors. Restore the early return for REPEATER_UNAUTHORIZED / REPEATER_NOT_PERMITTED; retrying an authorization rejection can't succeed. Renamed suppressConnectionError → isTerminalConnectionError, which is what it actually tests.
  • Make teardown authoritative. New disposed flag, set first in disconnect() and cleared in connect(). scheduleReconnection bails when set, as does the manual reconnect on io server disconnect.
  • Never dereference the socket through the throwing getter in deferred callbacks. The reconnection timer and deploy's process.nextTick now use this._socket?.…, so a late callback is a no-op instead of a fatal throw. deploy falls through to its existing No response. timeout, which callers already handle.
  • Fix the timer leak. clearConnectionTimer() at the top of scheduleReconnection keeps a single pending attempt; clearConnectionTimer now also resets the handle to undefined.
  • Fix the delay bounds so MIN (1s) < MAX (5s), which revives the backoff and makes the socket.io options coherent. The timer is deliberately left referenced so a pending attempt keeps a long-running Repeater alive during an outage.
  • Drop manager listeners on teardown. io() caches managers per URI, so without this a subsequent connect() would double-register the reserved-event handlers. Safe at that point: socket.disconnect() has already destroyed the socket and closed the manager.

Tests

New src/Repeater/DefaultRepeaterServer.spec.ts (9 cases), with socket.io-client mocked and fake timers.

On next without the source change, 6 of them fail — including two that fail with the exact Sentry message:

✕ should keep a single pending reconnection while attempts keep failing
✕ should not reconnect after a terminal repeater_unauthorized connection error
✕ should not reconnect after a terminal repeater_not_permitted connection error
✕ should not touch the socket when a pending reconnection fires after teardown
✕ should not schedule a reconnection when a connection error arrives during teardown
✕ should time out rather than throw when the socket is torn down before the deploy is flushed

Verification

  • npm run test:unit — 270 passed, 25 suites
  • npm run lint — clean
  • npm run format — clean
  • npm run build — webpack compiled successfully
  • npx tsc --noEmit — one pre-existing error in src/RequestExecutor/HttpRequestExecutor.spec.ts:158, present on next and unrelated to this change

Note, not addressed here

RepeaterServerEvents.ERROR is the string 'error' on a plain EventEmitter, so emitting it with no listener registered throws ERR_UNHANDLED_ERROR. Production is covered because ServerRepeaterLauncher.subscribeToEvents always registers one, but it's a sharp edge worth a separate look.

@aborovsky aborovsky self-assigned this Sep 11, 2026
@aborovsky aborovsky added the Type: bug Something isn't working. label Sep 11, 2026
@aborovsky

aborovsky commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Don't merge before testing locally
  • Ensure E2E tests passed for feature branch

@aborovsky
aborovsky enabled auto-merge (squash) September 16, 2026 07:38
@aborovsky
aborovsky merged commit 9e641a2 into next Sep 16, 2026
6 checks passed
@aborovsky
aborovsky deleted the fix/repeater-reconnect-after-teardown branch September 16, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type: bug Something isn't working.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants