Skip to content

Fix connection close ordering and authorize() waiting; un-gate UTS tests - #1364

Open
sacOO7 wants to merge 5 commits into
uts/realtime-uts-tests-and-infrafrom
fix/connection-ordering-and-authorize-wait
Open

sacOO7 wants to merge 5 commits into
uts/realtime-uts-tests-and-infrafrom
fix/connection-ordering-and-authorize-wait

Conversation

@sacOO7

@sacOO7 sacOO7 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR fixes three connection bugs that caused gated UTS tests to pass or fail depending on timing. It then un-gates the 21 UTS tests that now pass reliably.

# Fix Spec points User-visible?
1 A transport close is handled after the server message queued just before it, so a fatal error can no longer leave the connection CONNECTING after FAILED RTN14g, RTN15h3, RTN27h Yes: connection state after an error
2 authorize() subscribes to connection changes before sending AUTH, and waits for CONNECTED in every connection state RTC8a, RTC8b1, RTC8c Yes: authorize() waits instead of returning early or timing out
3 The shared ErrorInfo.NonRenewableToken is no longer modified in place RSA4a2 Yes: a later client could see an earlier client's error text

All three were found by triaging UTS tests that were gated ([DeviationFact]) but sometimes passed with RUN_DEVIATIONS=1. Each fix was cross-checked against ably-java and the features spec.

Stacked PR. The base is uts/realtime-uts-tests-and-infra (#1362), because the UTS tests touched here only exist on that branch. Once #1362 merges, retarget this PR to integration/v2.

Not in scope: issue #1361 (an operation completes before its public state-change event is emitted). The UTS tests that fail only because of that issue are untouched and stay gated; see "Still gated" below.


Changes

1. A transport close is handled after the message queued before it

Problem. When the server sends ERROR, DISCONNECTED or CONNECTED and then closes the transport, both events land in the realtime workflow queue back to back. Handling the message only queues a follow-up command, and that command lands behind the close. The close was therefore handled first, in the state the message was meant to leave. For a fatal ERROR the connection:

  1. disconnected;
  2. retried at once;
  3. went FAILED when the ERROR was finally handled;
  4. then ran the queued reconnect, ending up CONNECTING after FAILED.

Whether this happened depended on how quickly the workflow thread picked up the message, which is why the affected tests passed or failed at random.

Fix (Realtime/Workflows/RealtimeWorkflow.cs, RealtimeCommands.cs). A transport Closed or Exception event is re-queued once, so it is handled after the commands already queued. That restores the order in which the transport delivered them. The deferred close is then ignored if the transport has changed, or if the connection is already FAILED, DISCONNECTED or CLOSED.

Compared with ably-java. Same effect. Java decides the ERROR or DISCONNECTED target state inline on the socket thread and queues it ahead of the close's action (ConnectionManager.java around lines 1211, 1386–1411 and 1553–1591).

Side effects reviewed: normal client-initiated close (CLOSING → CLOSED), an exception with nothing queued, repeated closes, a close while CONNECTING, DISCONNECTED or SUSPENDED, heartbeat or idle disconnects (RTN23), a disposed client, and deadlocks. None regressed. On a disposed client the new order is slightly safer, because the old order could start a new transport after dispose.

2. authorize() waits for CONNECTED correctly

Problem (Transport/ConnectionManager.OnAuthUpdated):

  • While CONNECTED: it created its connection-state waiter after sending AUTH. A fast CONNECTED reply could arrive before the waiter existed, so authorize() waited the full timeout (10 s).
  • While not CONNECTED: it started a connection and returned at once, without waiting for CONNECTED (RTC8b1, RTC8c).

Fix.

  • ConnectionChangeAwaiter subscribes when it is constructed (Utils/ConnectionChangeAwaiter.cs).
  • OnAuthUpdated creates the waiter before it sends AUTH or starts a connection, and waits in every state.
  • The waiter unsubscribes on every path: success, failure, timeout and dispose.

Compared with ably-java. It matches: Java subscribes its waiter first and waits in every state (ConnectionManager.java around lines 1021–1090). One known difference remains: in CONNECTING, Java halts the pending attempt (RTC8b). That is a separate change and its tests stay gated.

REST retry kept as before. A REST request retried after a token error (RSA4b) renews the token but does not wait for the realtime connection, as in ably-java (Auth.java around lines 960–968). Without this, a REST request on a realtime client would block until the realtime connection was CONNECTED. This uses a new internal AuthorizeAsync(..., waitForConnection) overload in AblyAuth.cs, called from PubSubHttpClient.cs.

3. The shared NonRenewableToken error is no longer modified

Problem. ErrorInfo.NonRenewableToken is a single process-wide static readonly instance. On a non-renewable token error, the workflow appended the original error text to it (newError.Message += …). That text then leaked into every later use of that error, by any client.

Fix. The workflow now builds a new ErrorInfo with the combined message.


Public API

No public signature changes.

  • AuthorizeAsync: the public interface IAblyAuth is unchanged. AblyAuth.AuthorizeAsync(TokenParams, AuthOptions) keeps its exact signature (only the async keyword moved into a new internal overload).
  • Internal types only: every type touched in the SDK (AblyAuth, ConnectionManager, ConnectionChangeAwaiter, HandleTransportEventCommand, RealtimeWorkflow) is internal. In PubSubHttpClient only a method body changed.

The user-visible behaviour changes are listed in CHANGELOG.md under "Behaviour changes".


Tests

New native tests

Test Covers
RealtimeWorkflowSpecs+ConnectingStateSpecs.WithInboundErrorMessageFollowedByTransportClose_ShouldStayFailed (8 cases) Fix 1: ERROR/DISCONNECTED/CONNECTED followed by a close is handled in order
RealtimeWorkflowSpecs+ConnectingStateSpecs.SetFailedState_WithNonRenewableTokenError_ShouldNotChangeTheSharedNonRenewableTokenError Fix 3
ConnectionFailuresOnceConnectedSpecs.WhenNotConnected_AuthorizeShouldWaitForTheConnectionToBeConnected Fix 2
ConnectionFailuresOnceConnectedSpecs.WhenNotConnected_RestRequestRetriedAfterTokenError_ShouldNotWaitForTheRealtimeConnection Fix 2: the REST retry stays independent of the realtime connection

Each fails on the code before its fix.

UTS tests un-gated (21)

They were [DeviationFact], are now [UtsFact], and their deviations.md entries are removed or trimmed:

Class Tests
ChannelPublishTests RTL6c4_FailsConnFailed, RTN7e_ErrorRepresentsReason
ConnectionFailuresTests RTN15h2_TokenErrorRenewFails, RTN15h3_NonTokenErrorResume, RTN15j_ErrorEmptyChannelFailed
ConnectionOpenFailuresTests RTN14a_InvalidKeyFailed, RSA4a_TokenErrorNoRenewal
ConnectionPingTests RTN13b_DeferredPingErrorFailed
ErrorReasonTests RTN25_ErrorReasonOnFailed, RTN25_ErrorReasonTokenError, RTN25_ErrorReasonInStateChange
RealtimeAuthorizeTests RTC8a_AuthorizeConnectedSendsAuth, RTC8a1_CapabilityDowngradeChannelFailed, RTC8a3_AuthorizeCompletesAfterResponse, RTC8c_AuthorizeDisconnectedInitiatesConnection, RTC8c_AuthorizeClosedInitiatesConnection
RealtimePresenceGetTests RTP11c1_GetNoWaitReturnsImmediately
TokenExpiryNonRenewableTests RSA4a2_TokenErrorNonRenewableFailed, RSA4a2_TokenErrorNonRenewableNoRetry
Proxy ConnectionResumeTests RTN15h1_TokenErrorNonrenewableFailed
Infra StateWaitTests ConnectionStaysFailedWhenTheServerClosesAfterAFatalError

Two of these needed more than un-gating:

  • RTP11c1_GetNoWaitReturnsImmediately (test fix): it now drains pending events before get, as its own deviation entry suggested, with a // NOTE:.
  • ChannelDetachTests.RTL5l_DetachAttachedWhenDisconnected (now a UTS spec-error placeholder, skipped): the spec expects a DETACH while DISCONNECTED. But the SDK correctly reconnects at once after an unexpected drop (RTN15a), and detach() goes straight to DETACHED when not connected. It has a new UTS Spec Errors entry quoting the spec, with a draft fix.

Still gated

Reason Tests
Issue #1361 (event ordering, not fixed here) ChannelStateEventsTests: RTL2a_StateChangeEventsEmitted, RTL2_FilteredEventSubscription, RTL2d_ResumedFlagPropagated
ChannelConnectionStateTests: RTL3a_OtherStatesUnaffected, RTL3d_InitDetachedNotReattached
ChannelDetachTests: RTL5_DetachStateChangeEvents
ChannelErrorTests: RTL14_AttachedToFailed
RealtimeAuthorizeTests: RTC8a1_SuccessfulReauthUpdateEvent, RTC8a2_FailedReauthConnectionFailed, RTC8c_AuthorizeFailedInitiatesConnection
Proxy ChannelFaultsTests: RTL14_ChannelErrorGoesFailed, RTL12_AttachedNonResumedUpdate, RTL14_ErrorOnAttach
RTN14d (instant retry overwrites ErrorReason; follow-up) RTN25_ErrorReasonDisconnected, RTN25_ErrorReasonClearedOnConnect, RTN14c_ConnectionTimeout, RSA4c2_CallbackTimeoutConnectingDisconnected
RTC8b (halting a pending attempt; follow-up) RTC8b_AuthorizeConnectingHaltsAttempt, RTC8b1_AuthorizeConnectingFailsOnFailed

None of the #1361 tests' code was changed. In deviations.md, their entry now links the issue, and RTC8a2 / RTC8c_AuthorizeFailedInitiatesConnection moved into it: their other cause (the RTN27h race) is fixed here, and only the #1361 listener read still fails.


CI follow-up (second commit)

The first CI run of this PR failed two jobs. Neither failure was caused by the fixes above; both had already happened on the base branch:

Java cross-check: ably-java runs the attach() callback inside the state emit, over a copy of the listener list (ChannelBase.java around lines 176 and 262; EventEmitter.java around lines 143–147). So in Java a listener added after the callback never sees that event, which confirms the tests' expectations.

CI follow-up (third commit)

The next CI run failed only unit-netframework, on Uts.RestTests.Unit.FallbackTests.REC3_ConnectivityCheckValidation. This race was there before this PR. It also reproduces on the base branch with an added delay, and none of the code changed here is on its path.

  • Cause: the UTS spec queues one connectivity-check response and then builds an auto-connecting client with no WebSocket mock. The client's failed connection attempt runs the SDK's own RTN17j fallback connectivity check against the same URL, which can use up that single response. The test's own checkConnectivity() then times out.
  • Fix: the five REC3 clients use AutoConnect = false, each with a // NOTE:, recorded as an adapted UTS spec error.

The UTS spec-error classifications in this PR (REC3, RTL5l, RTP11c1) were independently re-verified against features.md and the UTS specs. All three are genuine spec errors, with the right treatment: REC3 and RTP11c1 adapted, RTL5l a skipped placeholder. The RTL5l entry was corrected to describe the actual race: an immediate RTN15a reconnect, after which RTL5i defers the detach to an ATTACH nothing answers. It now quotes RTL5i and proposes the spec's own refuse-reconnect fix.

CI follow-up (fourth commit)

The next CI run failed only unit-netframework, on the UTS infra self-test Uts.InfraTests.Unit.FakeClockTests.TimerHookIsPerClientAndAClientBuiltWithoutItKeepsTheRealClock ("Expected 1 … but found 2"). This race was there before this PR: RealtimeWorkflow.Start() and the heartbeat loop are identical on the base branch, and this test never calls authorize().

  • Cause: the fake-clock client puts two timers on the fake clock: the connecting-state timer, and the first RTN23a heartbeat monitor delay, which RealtimeWorkflow.Start registers from a Task.Run on a pool thread. The test counted pending timers right after the connection attempt. On a slow runner the heartbeat delay landed later, and was mistaken for a timer from the second client.
  • Fix (test only): wait until both of the first client's timers are scheduled before counting. The final assertion is unchanged, so the test still proves the second client adds no timer to the fake clock. Reproduced by injecting a 100 ms delay before the heartbeat loop: it fails without the fix and passes with it.

CI follow-up (fifth commit)

The next CI run failed proxy on Uts.RealtimeTests.Integration.Proxy.ConnectionOpenFailuresTests.RTN14c_ConnectionTimeout (Expected Disconnected, but found Connecting). It also failed integration, but only because the runner's NuGet restore hit CERTIFICATE_VERIFY_FAILED; no tests ran, and that job was re-run.

Validation

Run on net6.0, Release, in two configurations:

Check Result
./build.sh --target=Build.NetStandard (StyleCop as errors) 0 warnings, 0 errors
CI unit leg (Test.NetStandard.Unit.WithRetry) 1483 total: 1464 passed, 19 skipped, 0 failed, nothing retried
UTS unit tiers (REST + realtime + infra), plain 0 failed in both (a) and (b): 874 passed, 241 skipped
UTS unit tiers with RUN_DEVIATIONS=1 Every failure is a gated test in both configurations; no un-gated test failed
Each un-gated test run on its own Passes every run in both configurations (10× during development, 3× in the final pass)
UTS integration + proxy (sandbox) REST 70 passed, realtime 65 passed (including RTN15h1), 0 failed

Size

23 files, +383 / −239:

Area Files Lines
Product code 6 +175 / −77
Native tests 2 +135 / −0
UTS tests 11 +39 / −97
deviations.md 1 +27 / −63
CHANGELOG 1 +5 / −0
UTS infra notes (README, skill mapping) 2 +2 / −2

Reviewer notes and follow-ups

Summary by CodeRabbit

  • Bug Fixes
    • Realtime authorization now starts a connection when needed, including from initialized, closed, or disconnected states, and waits for the client to connect before completing. It reports an error if the connection fails.
    • REST requests retried after a token error can use refreshed credentials without waiting for the realtime connection.
    • Fatal errors followed by a socket close now leave the connection in the Failed state.
    • Non-renewable token errors retain their original details without changing the shared error message.

- Handle a transport close or exception after the commands already
  queued by the message before it. A server ERROR, DISCONNECTED or
  CONNECTED followed by a close was handled out of order, so a fatal
  error could leave the connection CONNECTING after FAILED (RTN14g,
  RTN15h3, RTN27h). Stop appending to the shared NonRenewableToken
  ErrorInfo.
- Make authorize() subscribe to connection changes before sending AUTH
  or connecting, and wait for CONNECTED in every connection state
  (RTC8a, RTC8b1, RTC8c). A REST request retried after a token error
  still renews the token without waiting for the realtime connection,
  as ably-java does.
- Add native tests, un-gate the 21 UTS tests these fixes resolve, turn
  RTL5l into a UTS spec-error placeholder, and update deviations.md,
  the CHANGELOG and the UTS infra notes. Tests that fail only because
  of the event-ordering issue stay gated.
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4743f105-4975-41ce-b8c8-9dce30efe52d
📥 Commits

Reviewing files that changed from the base of the PR and between e266c12 and 047c33a.

📒 Files selected for processing: 3
  • src/Ably.PubSub.Tests.Shared/Uts/InfraTests/Unit/FakeClockTests.cs
  • src/Ably.PubSub.Tests.Shared/Uts/RealtimeTests/Integration/Proxy/ConnectionOpenFailuresTests.cs
  • src/Ably.PubSub.Tests.Shared/Uts/deviations.md
🚧 Files skipped from review as they are similar to previous changes: 1
  • src/Ably.PubSub.Tests.Shared/Uts/deviations.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

Realtime authorization now waits for connection outcomes when requested, while REST token-error retries do not wait for the realtime connection. The workflow defers transport-close events behind earlier message commands and creates separate error details for non-renewable token failures. Related tests and UTS records are updated.

Changes

Realtime behavior and test coverage

Layer / File(s) Summary
Authorization connection waiting
src/Ably.PubSub.Shared/AblyAuth.cs, src/Ably.PubSub.Shared/Transport/ConnectionManager.cs, src/Ably.PubSub.Shared/Utils/ConnectionChangeAwaiter.cs, src/Ably.PubSub.Shared/PubSubHttpClient.cs, src/Ably.PubSub.Tests.Shared/Realtime/ConnectionSpecs/*, src/Ably.PubSub.Tests.Shared/Uts/RealtimeTests/Unit/RealtimeAuthorizeTests.cs, src/Ably.PubSub.Tests.Shared/Uts/deviations.md, CHANGELOG.md
Authorization waits for a connection outcome when enabled, including when it must initiate a connection. REST retries after token errors disable the wait. The awaiter queues connection-state changes, and tests cover both authorization paths.
Transport-close ordering and token errors
src/Ably.PubSub.Shared/Realtime/Workflows/RealtimeCommands.cs, src/Ably.PubSub.Shared/Realtime/Workflows/RealtimeWorkflow.cs, src/Ably.PubSub.Tests.Shared/Realtime/RealtimeWorkflowSpecs.cs, src/Ably.PubSub.Tests.Shared/Uts/InfraTests/StateWaitTests.cs, src/Ably.PubSub.Tests.Shared/Uts/Infra/README.md, .claude/skills/uts-to-csharp/references/realtime-mapping.md, CHANGELOG.md, src/Ably.PubSub.Tests.Shared/Uts/RealtimeTests/*
The workflow defers a close or exception event once so commands from earlier messages run first. Non-renewable token errors use a new ErrorInfo. Regression tests check the failed connection state and error details.
UTS test status and adaptations
src/Ably.PubSub.Tests.Shared/Uts/RealtimeTests/*, src/Ably.PubSub.Tests.Shared/Uts/deviations.md, src/Ably.PubSub.Tests.Shared/Uts/RestTests/Unit/FallbackTests.cs, src/Ably.PubSub.Tests.Shared/Uts/InfraTests/Unit/FakeClockTests.cs
Several UTS tests change between deviation-specific and standard attributes. RTP11c1 processes pending events before its no-wait read. RTL5l is represented by a skipped test. REC3 disables auto-connect in its test clients. Other changes update deviation records, proxy-tier test attributes, and a fake-clock timer assertion.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant AblyAuth
  participant ConnectionManager
  participant ConnectionChangeAwaiter
  participant RealtimeTransport
  Caller->>AblyAuth: AuthorizeAsync with connection waiting enabled
  AblyAuth->>ConnectionManager: OnAuthUpdated with wait enabled
  ConnectionManager->>ConnectionChangeAwaiter: Create waiter before connection attempt or AUTH
  ConnectionManager->>RealtimeTransport: Initiate connection or send AUTH
  RealtimeTransport-->>ConnectionChangeAwaiter: Report connection state changes
  ConnectionManager->>ConnectionChangeAwaiter: Wait for state changes
  ConnectionManager-->>AblyAuth: Complete after Connected
  AblyAuth-->>Caller: Return authorization result
Loading

Possibly related issues

Merge Risk: ⚪ Minimal · up to 047c3

No actionable merge risk remains from the reviewed changes.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 67.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 73 functions across 23 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly summarizes the main changes: fixing connection-close ordering, correcting authorize() waiting behavior, and enabling previously gated UTS tests.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 67.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 73 functions across 23 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the state-change queue
And waits until the link is true
A close steps back; old messages run
Fresh error details keep shared ones done
The tests now hop along with care

Comment @coderabbitai help to get the list of available commands.

…ordering issue

ChannelStateWaitLatchesAStateEnteredAndLeftBeforeItIsAwaited checked
IsCompleted right after AttachAsync, but the wait completes two
thread-pool hops later, so it failed on a busy CI runner. It now awaits
the wait with a timeout. RTL12_AttachedNonResumedUpdate and
RTL14_ErrorOnAttach fail when AttachAsync completes before its public
state event is emitted, which is the known event-ordering issue; gate
them and record them under that deviations.md entry.
… and correct the RTL5l spec-error entry

REC3_ConnectivityCheckValidation queued one connectivity-check response
and then built an auto-connecting client. Its failed connection attempt
ran the RTN17j fallback check against the same URL first, so the test's
own check could time out. The clients now use AutoConnect = false, and
the case is recorded as an adapted UTS spec error. The RTL5l entry and
placeholder comment now describe the real race (an immediate RTN15a
reconnect, then RTL5i deferring the detach), quote RTL5i, and propose
the spec's own refuse-reconnect fix.

This branch was successfully deployed

1 active deployment
staging/pull/1364/features — 047c33ac Deployed Oct 23, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant