Skip to content

fix: validate silo startup before runtime initialization - #35

Merged
Neftedollar merged 2 commits into
mainfrom
codex/fix-stream-expiry-release-gate
Sep 5, 2026
Merged

Neftedollar merged 2 commits into
mainfrom
codex/fix-stream-expiry-release-gate

Conversation

@Neftedollar

@Neftedollar Neftedollar commented Sep 5, 2026 •

Copy link
Copy Markdown
Owner

Summary

Follow-up to #34, covering the stream-expiry fixture and an independently diagnosed startup-validation leak exposed by the repeated CI runs.

Stream expiry

The repeated CI run on main exposed a timing assumption in the abandoned server-stream enumerator fixture:

  • The silo's 3-second response timeout is retained because Orleans uses it as the enumerator cleanup period.
  • The client gets a separate 30-second RPC budget. A new 4-second ordinary-call test prevents accidentally coupling that budget to expiry again.
  • The expiry test waits for the producer's cancellation/finally signal instead of assuming two timer periods plus a fixed margin have elapsed. This does not send another request or refresh the enumerator's seen flag.
  • A 45-second watchdog fails the test if cleanup never happens. The enumerator is asynchronously disposed even when an assertion fails.

Orleans 10.1.0 and 10.3.1 both remove the enumerator table entry before cancellation/disposal, so observing the producer's finally makes the subsequent missing-enumerator assertion meaningful.

Startup validation before runtime initialization

The initial PR run passed both expiry tests but timed out in an unrelated read-only gate test. A two-CPU full-suite reproduction then timed out in a journal test. Serializing xUnit collections did not fix it.

Managed stacks showed 13 busy Orleans DistributedGrainDirectory.ProcessMembershipUpdates loops. A separate C# probe, referencing only stock Microsoft Orleans 10.3.1, reproduced the cause: a failed observer in RuntimeInitialize can leave runtime services started in that same stage running after disposal. The directory's membership stream completes while its stop token remains uncancelled, and the loop repeatedly subscribes to the completed stream.

The functional silo validator now runs at ServiceLifecycleStage.First. Its checks need registered services/options and the local manifest, not a running silo. Invalid definitions therefore fail before the runtime-start stage. The real-lifecycle regression asserts that a RuntimeInitialize sentinel is never entered on rejection and is entered on valid startup. Documentation, its website mirror, the LLM corpus, and the changelog describe the earlier validation boundary.

Verification

  • Reproduced the old client configuration: the 4-second RPC fails with Response did not arrive on time in 00:00:03.
  • Fixed focused tests: 2/2 passed on Orleans 10.3.1.
  • Negative control: temporarily disabling expiry makes the expiry test fail its 45-second watchdog; normal expiry was restored afterwards.
  • Startup-order regression before the fix: invalid arm failed with expected 0, actual 1 runtime starts; valid arm passed.
  • All 15 hosting tests passed after the fix.
  • Original two-CPU CI filter, with collection parallelism unchanged: 285 passed, one expected Redis skip after the fix (286 total).
  • Release unit tests: 2153 passed on each of Orleans 10.1.0 and 10.3.1; warnings are treated as errors.
  • Full local Release integration: Orleans 10.1.0 — 588 passed and 11 expected skips; Orleans 10.3.1 — 589 passed and 10 expected skips (599 total in each). Redis is checked separately by the mandatory CI jobs.
  • Docs mirror, current-API/deprecation checks, and generated LLM corpus are consistent.

No public package API, package version, tag, or publication changes. No test assertions or timeouts were weakened to address the startup leak, and collection parallelism remains enabled.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-05T20:31:00.671741Z d8b98c0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Neftedollar Neftedollar changed the title test: separate stream expiry from client RPC timeouts fix: validate silo startup before runtime initialization Sep 5, 2026
@Neftedollar
Neftedollar merged commit a57de69 into main Sep 5, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant