Repository navigation
Conversation
- Handle a transport close or exception after the commands already queued by the message before it. A server ERROR, DISCONNECTED or CONNECTED followed by a close was handled out of order, so a fatal error could leave the connection CONNECTING after FAILED (RTN14g, RTN15h3, RTN27h). Stop appending to the shared NonRenewableToken ErrorInfo. - Make authorize() subscribe to connection changes before sending AUTH or connecting, and wait for CONNECTED in every connection state (RTC8a, RTC8b1, RTC8c). A REST request retried after a token error still renews the token without waiting for the realtime connection, as ably-java does. - Add native tests, un-gate the 21 UTS tests these fixes resolve, turn RTL5l into a UTS spec-error placeholder, and update deviations.md, the CHANGELOG and the UTS infra notes. Tests that fail only because of the event-ordering issue stay gated.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughRealtime authorization now waits for connection outcomes when requested, while REST token-error retries do not wait for the realtime connection. The workflow defers transport-close events behind earlier message commands and creates separate error details for non-renewable token failures. Related tests and UTS records are updated. ChangesRealtime behavior and test coverage
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant Caller
participant AblyAuth
participant ConnectionManager
participant ConnectionChangeAwaiter
participant RealtimeTransport
Caller->>AblyAuth: AuthorizeAsync with connection waiting enabled
AblyAuth->>ConnectionManager: OnAuthUpdated with wait enabled
ConnectionManager->>ConnectionChangeAwaiter: Create waiter before connection attempt or AUTH
ConnectionManager->>RealtimeTransport: Initiate connection or send AUTH
RealtimeTransport-->>ConnectionChangeAwaiter: Report connection state changes
ConnectionManager->>ConnectionChangeAwaiter: Wait for state changes
ConnectionManager-->>AblyAuth: Complete after Connected
AblyAuth-->>Caller: Return authorization result
Possibly related issues
Merge Risk: ⚪ Minimal · up to
|
…ordering issue ChannelStateWaitLatchesAStateEnteredAndLeftBeforeItIsAwaited checked IsCompleted right after AttachAsync, but the wait completes two thread-pool hops later, so it failed on a busy CI runner. It now awaits the wait with a timeout. RTL12_AttachedNonResumedUpdate and RTL14_ErrorOnAttach fail when AttachAsync completes before its public state event is emitted, which is the known event-ordering issue; gate them and record them under that deviations.md entry.
… and correct the RTL5l spec-error entry REC3_ConnectivityCheckValidation queued one connectivity-check response and then built an auto-connecting client. Its failed connection attempt ran the RTN17j fallback check against the same URL first, so the test's own check could time out. The clients now use AutoConnect = false, and the case is recorded as an adapted UTS spec error. The RTL5l entry and placeholder comment now describe the real race (an immediate RTN15a reconnect, then RTL5i deferring the detach), quote RTL5i, and propose the spec's own refuse-reconnect fix.
…ck in the timer-hook self-test
…econnect deviation
Summary
This PR fixes three connection bugs that caused gated UTS tests to pass or fail depending on timing. It then un-gates the 21 UTS tests that now pass reliably.
authorize()subscribes to connection changes before sending AUTH, and waits for CONNECTED in every connection stateauthorize()waits instead of returning early or timing outErrorInfo.NonRenewableTokenis no longer modified in placeAll three were found by triaging UTS tests that were gated (
[DeviationFact]) but sometimes passed withRUN_DEVIATIONS=1. Each fix was cross-checked against ably-java and the features spec.Changes
1. A transport close is handled after the message queued before it
Problem. When the server sends ERROR, DISCONNECTED or CONNECTED and then closes the transport, both events land in the realtime workflow queue back to back. Handling the message only queues a follow-up command, and that command lands behind the close. The close was therefore handled first, in the state the message was meant to leave. For a fatal ERROR the connection:
Whether this happened depended on how quickly the workflow thread picked up the message, which is why the affected tests passed or failed at random.
Fix (
Realtime/Workflows/RealtimeWorkflow.cs,RealtimeCommands.cs). A transport Closed or Exception event is re-queued once, so it is handled after the commands already queued. That restores the order in which the transport delivered them. The deferred close is then ignored if the transport has changed, or if the connection is already FAILED, DISCONNECTED or CLOSED.Compared with ably-java. Same effect. Java decides the ERROR or DISCONNECTED target state inline on the socket thread and queues it ahead of the close's action (
ConnectionManager.javaaround lines 1211, 1386–1411 and 1553–1591).Side effects reviewed: normal client-initiated close (CLOSING → CLOSED), an exception with nothing queued, repeated closes, a close while CONNECTING, DISCONNECTED or SUSPENDED, heartbeat or idle disconnects (RTN23), a disposed client, and deadlocks. None regressed. On a disposed client the new order is slightly safer, because the old order could start a new transport after dispose.
2.
authorize()waits for CONNECTED correctlyProblem (
Transport/ConnectionManager.OnAuthUpdated):authorize()waited the full timeout (10 s).Fix.
ConnectionChangeAwaitersubscribes when it is constructed (Utils/ConnectionChangeAwaiter.cs).OnAuthUpdatedcreates the waiter before it sends AUTH or starts a connection, and waits in every state.Compared with ably-java. It matches: Java subscribes its waiter first and waits in every state (
ConnectionManager.javaaround lines 1021–1090). One known difference remains: in CONNECTING, Java halts the pending attempt (RTC8b). That is a separate change and its tests stay gated.REST retry kept as before. A REST request retried after a token error (RSA4b) renews the token but does not wait for the realtime connection, as in ably-java (
Auth.javaaround lines 960–968). Without this, a REST request on a realtime client would block until the realtime connection was CONNECTED. This uses a new internalAuthorizeAsync(..., waitForConnection)overload inAblyAuth.cs, called fromPubSubHttpClient.cs.3. The shared
NonRenewableTokenerror is no longer modifiedProblem.
ErrorInfo.NonRenewableTokenis a single process-widestatic readonlyinstance. On a non-renewable token error, the workflow appended the original error text to it (newError.Message += …). That text then leaked into every later use of that error, by any client.Fix. The workflow now builds a new
ErrorInfowith the combined message.Public API
No public signature changes.
AuthorizeAsync: the public interfaceIAblyAuthis unchanged.AblyAuth.AuthorizeAsync(TokenParams, AuthOptions)keeps its exact signature (only theasynckeyword moved into a newinternaloverload).AblyAuth,ConnectionManager,ConnectionChangeAwaiter,HandleTransportEventCommand,RealtimeWorkflow) isinternal. InPubSubHttpClientonly a method body changed.The user-visible behaviour changes are listed in
CHANGELOG.mdunder "Behaviour changes".Tests
New native tests
RealtimeWorkflowSpecs+ConnectingStateSpecs.WithInboundErrorMessageFollowedByTransportClose_ShouldStayFailed(8 cases)RealtimeWorkflowSpecs+ConnectingStateSpecs.SetFailedState_WithNonRenewableTokenError_ShouldNotChangeTheSharedNonRenewableTokenErrorConnectionFailuresOnceConnectedSpecs.WhenNotConnected_AuthorizeShouldWaitForTheConnectionToBeConnectedConnectionFailuresOnceConnectedSpecs.WhenNotConnected_RestRequestRetriedAfterTokenError_ShouldNotWaitForTheRealtimeConnectionEach fails on the code before its fix.
UTS tests un-gated (21)
They were
[DeviationFact], are now[UtsFact], and theirdeviations.mdentries are removed or trimmed:ChannelPublishTestsRTL6c4_FailsConnFailed,RTN7e_ErrorRepresentsReasonConnectionFailuresTestsRTN15h2_TokenErrorRenewFails,RTN15h3_NonTokenErrorResume,RTN15j_ErrorEmptyChannelFailedConnectionOpenFailuresTestsRTN14a_InvalidKeyFailed,RSA4a_TokenErrorNoRenewalConnectionPingTestsRTN13b_DeferredPingErrorFailedErrorReasonTestsRTN25_ErrorReasonOnFailed,RTN25_ErrorReasonTokenError,RTN25_ErrorReasonInStateChangeRealtimeAuthorizeTestsRTC8a_AuthorizeConnectedSendsAuth,RTC8a1_CapabilityDowngradeChannelFailed,RTC8a3_AuthorizeCompletesAfterResponse,RTC8c_AuthorizeDisconnectedInitiatesConnection,RTC8c_AuthorizeClosedInitiatesConnectionRealtimePresenceGetTestsRTP11c1_GetNoWaitReturnsImmediatelyTokenExpiryNonRenewableTestsRSA4a2_TokenErrorNonRenewableFailed,RSA4a2_TokenErrorNonRenewableNoRetryConnectionResumeTestsRTN15h1_TokenErrorNonrenewableFailedStateWaitTestsConnectionStaysFailedWhenTheServerClosesAfterAFatalErrorTwo of these needed more than un-gating:
RTP11c1_GetNoWaitReturnsImmediately(test fix): it now drains pending events beforeget, as its own deviation entry suggested, with a// NOTE:.ChannelDetachTests.RTL5l_DetachAttachedWhenDisconnected(now a UTS spec-error placeholder, skipped): the spec expects a DETACH while DISCONNECTED. But the SDK correctly reconnects at once after an unexpected drop (RTN15a), anddetach()goes straight to DETACHED when not connected. It has a new UTS Spec Errors entry quoting the spec, with a draft fix.Still gated
ChannelStateEventsTests:RTL2a_StateChangeEventsEmitted,RTL2_FilteredEventSubscription,RTL2d_ResumedFlagPropagatedChannelConnectionStateTests:RTL3a_OtherStatesUnaffected,RTL3d_InitDetachedNotReattachedChannelDetachTests:RTL5_DetachStateChangeEventsChannelErrorTests:RTL14_AttachedToFailedRealtimeAuthorizeTests:RTC8a1_SuccessfulReauthUpdateEvent,RTC8a2_FailedReauthConnectionFailed,RTC8c_AuthorizeFailedInitiatesConnectionProxy
ChannelFaultsTests:RTL14_ChannelErrorGoesFailed,RTL12_AttachedNonResumedUpdate,RTL14_ErrorOnAttachErrorReason; follow-up)RTN25_ErrorReasonDisconnected,RTN25_ErrorReasonClearedOnConnect,RTN14c_ConnectionTimeout,RSA4c2_CallbackTimeoutConnectingDisconnectedRTC8b_AuthorizeConnectingHaltsAttempt,RTC8b1_AuthorizeConnectingFailsOnFailedNone of the #1361 tests' code was changed. In
deviations.md, their entry now links the issue, andRTC8a2/RTC8c_AuthorizeFailedInitiatesConnectionmoved into it: their other cause (the RTN27h race) is fixed here, and only the #1361 listener read still fails.CI follow-up (second commit)
The first CI run of this PR failed two jobs. Neither failure was caused by the fixes above; both had already happened on the base branch:
unit (net6.0):InfraTests.StateWaitTests.ChannelStateWaitLatchesAStateEnteredAndLeftBeforeItIsAwaited. The test checkedIsCompletedright afterAttachAsync, but the wait completes two thread-pool hops later, so it raced on a busy runner. It now awaits the wait with a timeout.proxy:ChannelFaultsTests.RTL12_AttachedNonResumedUpdate. This is issue Channel AttachAsync/DetachAsync and AuthorizeAsync complete before the public state-change event is emitted (no SynchronizationContext) #1361: a listener registered right afterAttachAsync()still receives the attach's own ATTACHED. It also failed the same way on the base branch's CI. Now gated under the Channel AttachAsync/DetachAsync and AuthorizeAsync complete before the public state-change event is emitted (no SynchronizationContext) #1361 entry, together with its FAILED counterpartRTL14_ErrorOnAttach, which fails the same way without a synchronization context.Java cross-check: ably-java runs the
attach()callback inside the state emit, over a copy of the listener list (ChannelBase.javaaround lines 176 and 262;EventEmitter.javaaround lines 143–147). So in Java a listener added after the callback never sees that event, which confirms the tests' expectations.CI follow-up (third commit)
The next CI run failed only
unit-netframework, onUts.RestTests.Unit.FallbackTests.REC3_ConnectivityCheckValidation. This race was there before this PR. It also reproduces on the base branch with an added delay, and none of the code changed here is on its path.checkConnectivity()then times out.AutoConnect = false, each with a// NOTE:, recorded as an adapted UTS spec error.The UTS spec-error classifications in this PR (REC3, RTL5l, RTP11c1) were independently re-verified against
features.mdand the UTS specs. All three are genuine spec errors, with the right treatment: REC3 and RTP11c1 adapted, RTL5l a skipped placeholder. The RTL5l entry was corrected to describe the actual race: an immediate RTN15a reconnect, after which RTL5i defers the detach to an ATTACH nothing answers. It now quotes RTL5i and proposes the spec's own refuse-reconnect fix.CI follow-up (fourth commit)
The next CI run failed only
unit-netframework, on the UTS infra self-testUts.InfraTests.Unit.FakeClockTests.TimerHookIsPerClientAndAClientBuiltWithoutItKeepsTheRealClock("Expected 1 … but found 2"). This race was there before this PR:RealtimeWorkflow.Start()and the heartbeat loop are identical on the base branch, and this test never callsauthorize().RealtimeWorkflow.Startregisters from aTask.Runon a pool thread. The test counted pending timers right after the connection attempt. On a slow runner the heartbeat delay landed later, and was mistaken for a timer from the second client.CI follow-up (fifth commit)
The next CI run failed
proxyonUts.RealtimeTests.Integration.Proxy.ConnectionOpenFailuresTests.RTN14c_ConnectionTimeout(Expected Disconnected, but found Connecting). It also failedintegration, but only because the runner's NuGet restore hitCERTIFICATE_VERIFY_FAILED; no tests ran, and that job was re-run.CheckInstantRetryFlagretries at once even with no fallback host, so DISCONNECTED lasts only a moment and the spec's state check after the wait can see CONNECTING. The code involved is unchanged by this PR, and the unit copy of the test was already gated for the same reason.[DeviationFact("RTN14c")]with its assertions unchanged, and thedeviations.mdRTN14d entry lists both tiers.Validation
Run on net6.0, Release, in two configurations:
SynchronizationContext(parallelizeTestCollections=false). That is how console and server apps run, and it's the configuration that exposes Channel AttachAsync/DetachAsync and AuthorizeAsync complete before the public state-change event is emitted (no SynchronizationContext) #1361../build.sh --target=Build.NetStandard(StyleCop as errors)Test.NetStandard.Unit.WithRetry)RUN_DEVIATIONS=1RTN15h1), 0 failedSize
23 files, +383 / −239:
deviations.mdReviewer notes and follow-ups
authorize()now waits for CONNECTED in every state;disconnectedRetryTimeout, and the next CONNECTING overwritesErrorReason. After a failed connection attempt with no fallback host, the client reconnects at once instead of waiting disconnectedRetryTimeout (RTN14d) #1365authorize()while CONNECTING doesn't halt the pending attempt. authorize() while CONNECTING does not halt the pending connection attempt or reconnect with the new token (RTC8b) #1366authorize()fails with 40140 afterrealtimeRequestTimeoutwhile the connection is still DISCONNECTED and retrying. authorize() fails with 40140 after realtimeRequestTimeout while the connection is still DISCONNECTED and retrying (RTC8b1) #1367ErrorInfo.ReasonDisconnectedis mutated as a shared static, the same class of bug as fix 3. Shared static ErrorInfo.ReasonDisconnected is mutated with each client's transport exception #1368*.WithRetrytargets exit 0 when a test fails on both the run and the retry. Cake *.WithRetry test targets exit 0 when a test fails on both the run and the retry #1369Authorize()called from a state-change handler with no sync context can block for up to 10 s outside CONNECTED. ably-java behaves the same way.RUN_DEVIATIONSenables gated tests for any non-empty value, including0.Summary by CodeRabbit