Skip to content

feat(core): resolve run.returnValue via a World long poll instead of a 1s poll - #3570

Open
pranaygp wants to merge 3 commits into
mainfrom
pgp/run-status-long-poll
Open

feat(core): resolve run.returnValue via a World long poll instead of a 1s poll#3570
pranaygp wants to merge 3 commits into
mainfrom
pgp/run-status-long-poll

Conversation

@pranaygp

Copy link
Copy Markdown
Contributor

Motivation

await run.returnValue polls the run record on a fixed ~1s interval (Run#pollReturnValue, packages/core/src/runtime/run.ts). A run that finishes just after one of those reads is reported to the caller up to a full second late — quantization latency on a run that is already done, paid on every awaited run: a parent awaiting a child workflow, wf run --wait, a request handler awaiting a result.

This replaces #3023, which closed that gap client-side by having the completing invocation write a marker to a run-scoped stream. Per the review:

This is clever, but I'd like it not to be clever. We end up having to ensure parity of any client-side originated workflow termination source and the stream, which has a lot of edge-cases like server-originated termination of runs, aside from adding a stream to every single run that might not even be used.

Can't we have await run.returnValue call a world endpoint that long-polls for a run return status, which the world implementation can then simply have an endpoint for? […] it's backwards compatible with run.returnValue being able to fall back to a poll if the long-poll endpoint isn't available

So: no client-side signal, no per-run stream, no terminal-transition parity table. The World is asked to wait, and where it can't, nothing changes.

Server half: vercel/workflow-server#785 (the GET /v2/runs/:runId/status route).

The World API

// Storage['runs'], optional
waitForTerminalStatus?(
  id: string,
  params?: { timeoutMs?: number; signal?: AbortSignal; resolveData?: 'all' | 'none' }
): Promise<WorkflowRun>;

The contract, in full:

  • Resolve as soon as the run is terminal, with the same entity get returns.
  • Resolve no later than roughly timeoutMs with the latest snapshot, whatever its status. A timeout is a normal return, never an error — a still-running run is a legitimate answer.
  • timeoutMs is an upper bound, not a lower one. An implementation may resolve early with a non-terminal snapshot (world-vercel does when the backend has no long-poll route), so callers pace their own retries.
  • Fail exactly like get — a missing run throws WorkflowRunNotFoundError.

Nothing is declared beyond the method's presence: no capability flag, no env var to enable. A World that can't wait omits it and the runtime keeps interval-polling get.

The runtime loop

#pollReturnValue keeps its exact shape — the wait just replaces get + sleep:

const run = waitForTerminalStatus
  ? await waitForTerminalStatus(this.runId, { timeoutMs: getReturnValueWaitTimeoutMs() })
  : await world.runs.get(this.runId);
// … the same completed / cancelled / failed switch, unchanged …

and the sleep in the not-yet-terminal branch becomes "sleep off whatever is left of one poll interval":

const remainingIntervalMs = getReturnValuePollIntervalMs() - (Date.now() - iterationStartedAt);
if (remainingIntervalMs > 0) await sleep(remainingIntervalMs);

One expression, three jobs: it is the familiar fixed 1s sleep on the plain-poll path; a no-op after a wait that already blocked longer than the interval; and an anti-spin floor for a World whose wait returns early — that World degrades to interval polling rather than hot-looping. Resolved and rejected values are unchanged: same hydrateWorkflowReturnValue, same WorkflowRunCancelledError / WorkflowRunFailedError (with errorCode and hydrated cause), same resilientStart 404 retry ladder.

WORKFLOW_RETURN_VALUE_LONG_POLL=0 (or false) is the kill switch and restores the fixed-interval poll exactly. WORKFLOW_RETURN_VALUE_WAIT_MS tunes the per-call budget (default 25s — comfortably inside world-vercel's 60s per-request HTTP timeout, so the budget always expires as a response, not as a client timeout).

Per-world implementations

Each World waits with whatever its store already offers. All three re-read the run before answering — the notification is a signal only, never a source of truth — and back the wait with a periodic re-read, so a lost notification costs latency rather than a hang.

World Mechanism Backstop Verified
world-vercel long poll: GET /v2/runs/:runId/status?waitMs=… (vercel/workflow-server#785) server-side, 1s unit (undici MockAgent, 8 cases)
world-postgres LISTEN/NOTIFY on workflow_run_status, published after each run-terminal UPDATE; one lazily-opened shared listener connection WORKFLOW_POSTGRES_RUN_STATUS_POLL_INTERVAL_MS, 1s real Postgres via testcontainers, 8 cases
world-local in-process emitter signalled from writeRunUnderLifecycleLock (the one helper every run-lifecycle write funnels through) WORKFLOW_LOCAL_RUN_STATUS_POLL_INTERVAL_MS, 100ms — also covers multi-process dev over one data dir real fs-backed storage, 7 cases
world-sim not implemented — deliberate: a real wait would stall the simulator's virtual clock n/a falls back to interval polling

Degradation, in three layers

  1. World omits the method → interval-poll get, byte-identical to today.
  2. world-vercel's backend has no route. A 404 there is ambiguous — the run may not exist, or the server may predate the route — so it is resolved by falling back to the plain read, which is the answer we want either way: it raises WorkflowRunNotFoundError for a missing run, and returns the run when the route was what was missing. Only the latter (proof that the run exists and the route does not) marks the fast path unsupported, so one bad run ID can never disable long polling for the process. The mark expires after 5 minutes so a client that outlives a server roll-forward picks the fast path back up on its own. 405/501 are treated the same way; a 500 propagates untouched rather than being masked by a second read.
  3. Wait returns early / budget clamps to zero → the loop's interval floor paces it, i.e. today's behavior.

The budget is also clamped to leave 10s under the adapter's per-request HTTP timeout, so a long poll can never be the thing that times out.

Tests

  • packages/core/src/runtime/run-return-value-long-poll.test.ts (16 cases, fake timers) — uses the long poll and doesn't touch runs.get; forwards the configured budget; hydrates a completed return value; paces a World whose wait returns non-terminal early (asserts no second attempt before the interval, and one immediately after); no added sleep when the wait already outlasted the interval; interval-polls get when the method is absent; kill switch restores the fixed poll and never calls the wait; env parsing for both new vars.
  • packages/world-vercel/src/run-status-long-poll.test.ts (8 cases) — path/params incl. lazy refs and budget clamping; non-terminal snapshot returned as a value; no-budget → plain read; missing route degrades and stays degraded (one status attempt across two calls); missing run → WorkflowRunNotFoundError with long polling still enabled; 500 propagates.
  • packages/world-postgres/test/run-status-wait.test.ts (8 cases, testcontainers Postgres) — with the backstop dialed to 10s, so a wait that resolves in milliseconds proves the NOTIFY did the waking; completed / failed / cancelled; budget expiry; abort; unknown run; and a storage built without the listener still resolving on its re-read.
  • packages/world-local/src/storage/run-status-wait.test.ts (7 cases, real fs) — already-terminal, wake on completion and on cancellation, budget expiry, no budget, abort, unknown run.

Test plan

  • packages/core unit suite — 2131 passed, 3 expected-fail (quickjs-runtime.test.ts needs generated assets; unrelated, fails on main in this sandbox)
  • @workflow/world-vercel — 509 passed
  • @workflow/world-local — 549 passed (4 pre-existing chmod-based failures, also failing on main here)
  • @workflow/world-postgres — 174 passed (test/spec.test.ts needs a built @workflow/world-testing; unrelated)
  • tsc --noEmit clean: world, world-vercel, world-local, world-postgres, core
  • biome check clean on changed files
  • E2E against a preview deployment pointed at vercel/workflow-server#785

Docs

world.runs.waitForTerminalStatus() in the World storage reference, and an "Optional: Waiting for a Terminal Run Status" section in Building a World — the contract, the three reference mechanisms, and why omitting it is a supported choice.

🤖 Generated with Claude Code

`await run.returnValue` re-reads the run every second, so a run that finishes
just after a read is reported up to a full `WORKFLOW_RETURN_VALUE_POLL_INTERVAL_MS`
late — quantization latency on a run that is already done.

Add an optional `Storage['runs'].waitForTerminalStatus(runId, { timeoutMs,
signal, resolveData })`: one read the World holds open until the run reaches a
terminal status, returning the same entity `runs.get()` returns. An expired
budget returns the latest snapshot rather than throwing, so `#pollReturnValue`
keeps its shape — the wait simply replaces `get` + sleep in the loop, and a
terminal status is observed the moment it lands.

Implemented three ways, because "wait" means something different per store:
world-vercel long polls workflow-server's new `GET /v2/runs/:runId/status`
route; world-postgres parks on a `LISTEN`/`NOTIFY` published by its
run-terminal writes; world-local uses an in-process emitter over its run
files. All three re-read the run before answering and back the wait with a
periodic re-read, so a lost notification costs latency, never correctness.

The fast path is strictly additive. A World that omits the method (world-sim,
third-party adapters) keeps interval-polling `get` exactly as before, and
world-vercel falls back to the plain read when the workflow-server it is
talking to has no such route — telling a missing route from a missing run by
whether that read succeeds. The loop also keeps consecutive non-terminal
observations one poll interval apart, so a World whose wait returns early
degrades to polling instead of spinning.
`WORKFLOW_RETURN_VALUE_LONG_POLL=0` restores the fixed-interval poll.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 14, 2026 23:32
@pranaygp
pranaygp requested review from a team, fantix and msullivan as code owners August 14, 2026 23:32
@changeset-bot

changeset-bot Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: b85dbfc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages
Name Type
@workflow/core Minor
@workflow/world Minor
@workflow/world-local Minor
@workflow/world-postgres Minor
@workflow/world-vercel Minor
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Minor
@workflow/world-testing Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview Aug 15, 2026 12:16am
example-nextjs-workflow-webpack Ready Ready Preview Aug 15, 2026 12:16am
example-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-astro-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-express-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-fastify-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-hono-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-nestjs-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-nitro-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-nuxt-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-python-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-sveltekit-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-tanstack-start-workflow Ready Ready Preview Aug 15, 2026 12:16am
workbench-vite-workflow Ready Ready Preview Aug 15, 2026 12:16am
workflow-docs Ready Ready Preview, v0 Aug 15, 2026 12:16am
workflow-swc-playground Ready Ready Preview Aug 15, 2026 12:16am
workflow-tarballs Ready Ready Preview Aug 15, 2026 12:16am
workflow-web Ready Ready Preview Aug 15, 2026 12:16am

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • completes but callbacks are not called (GAP) (nextjs-webpack)
  • maxRetries=0 disables retries (nextjs-webpack)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 00:17:46Z · abandoned wrun_01M01CFV98PHE8SDS4PXDA2YN8

E2E Test Summary

Summary
Passed Failed Skipped Total
✅ ▲ Vercel Production 3321 0 735 4056
✅ 💻 Local Development 3673 0 539 4212
✅ 📦 Local Production 3810 0 558 4368
✅ 🐘 Local Postgres 3810 0 558 4368
✅ 🪟 Windows 312 0 0 312
✅ 🌐 Cross-language Conformance 9 0 128 137
✅ vercel-multi-region 27 0 0 27
Total 14962 0 2518 17480
Details by Category

✅ ▲ Vercel Production

App Passed Failed Skipped
✅ astro-node 128 0 28
✅ astro-quickjs 128 0 28
✅ example-node 128 0 28
✅ example-quickjs 128 0 28
✅ express-node 128 0 28
✅ express-quickjs 128 0 28
✅ fastify-node 128 0 28
✅ fastify-quickjs 128 0 28
✅ hono-node 128 0 28
✅ hono-quickjs 128 0 28
✅ nest-node 128 0 28
✅ nest-quickjs 128 0 28
✅ nextjs-turbopack-node 153 0 3
✅ nextjs-webpack-node 153 0 3
✅ nextjs-webpack-quickjs 153 0 3
✅ nitro-node 128 0 28
✅ nitro-quickjs 128 0 28
✅ nuxt-node 128 0 28
✅ nuxt-quickjs 128 0 28
✅ python-node 8 0 148
✅ sveltekit-node 147 0 9
✅ sveltekit-quickjs 147 0 9
✅ tanstack-start-node 128 0 28
✅ tanstack-start-quickjs 128 0 28
✅ vite-node 128 0 28
✅ vite-quickjs 128 0 28

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
✅ astro-stable-quickjs 130 0 26
✅ express-stable-node 130 0 26
✅ express-stable-quickjs 130 0 26
✅ fastify-stable-node 130 0 26
✅ fastify-stable-quickjs 130 0 26
✅ hono-stable-node 130 0 26
✅ hono-stable-quickjs 130 0 26
✅ nest-stable-node 130 0 26
✅ nest-stable-quickjs 130 0 26
✅ nextjs-turbopack-canary-node 137 0 19
✅ nextjs-turbopack-canary-quickjs 137 0 19
✅ nextjs-turbopack-stable-node 156 0 0
✅ nextjs-turbopack-stable-quickjs 156 0 0
✅ nextjs-webpack-canary-quickjs 137 0 19
✅ nextjs-webpack-stable-node 156 0 0
✅ nextjs-webpack-stable-quickjs 156 0 0
✅ nitro-stable-node 130 0 26
✅ nitro-stable-quickjs 130 0 26
✅ nuxt-stable-node 130 0 26
✅ nuxt-stable-quickjs 130 0 26
✅ sveltekit-stable-node 149 0 7
✅ sveltekit-stable-quickjs 149 0 7
✅ tanstack-start-node 130 0 26
✅ tanstack-start-quickjs 130 0 26
✅ vite-stable-node 130 0 26
✅ vite-stable-quickjs 130 0 26

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
✅ astro-stable-quickjs 130 0 26
✅ express-stable-node 130 0 26
✅ express-stable-quickjs 130 0 26
✅ fastify-stable-node 130 0 26
✅ fastify-stable-quickjs 130 0 26
✅ hono-stable-node 130 0 26
✅ hono-stable-quickjs 130 0 26
✅ nest-stable-node 130 0 26
✅ nest-stable-quickjs 130 0 26
✅ nextjs-turbopack-canary-node 137 0 19
✅ nextjs-turbopack-canary-quickjs 137 0 19
✅ nextjs-turbopack-stable-node 156 0 0
✅ nextjs-turbopack-stable-quickjs 156 0 0
✅ nextjs-webpack-canary-node 137 0 19
✅ nextjs-webpack-canary-quickjs 137 0 19
✅ nextjs-webpack-stable-node 156 0 0
✅ nextjs-webpack-stable-quickjs 156 0 0
✅ nitro-stable-node 130 0 26
✅ nitro-stable-quickjs 130 0 26
✅ nuxt-stable-node 130 0 26
✅ nuxt-stable-quickjs 130 0 26
✅ sveltekit-stable-node 149 0 7
✅ sveltekit-stable-quickjs 149 0 7
✅ tanstack-start-node 130 0 26
✅ tanstack-start-quickjs 130 0 26
✅ vite-stable-node 130 0 26
✅ vite-stable-quickjs 130 0 26

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
✅ astro-stable-quickjs 130 0 26
✅ express-stable-node 130 0 26
✅ express-stable-quickjs 130 0 26
✅ fastify-stable-node 130 0 26
✅ fastify-stable-quickjs 130 0 26
✅ hono-stable-node 130 0 26
✅ hono-stable-quickjs 130 0 26
✅ nest-stable-node 130 0 26
✅ nest-stable-quickjs 130 0 26
✅ nextjs-turbopack-canary-node 137 0 19
✅ nextjs-turbopack-canary-quickjs 137 0 19
✅ nextjs-turbopack-stable-node 156 0 0
✅ nextjs-turbopack-stable-quickjs 156 0 0
✅ nextjs-webpack-canary-node 137 0 19
✅ nextjs-webpack-canary-quickjs 137 0 19
✅ nextjs-webpack-stable-node 156 0 0
✅ nextjs-webpack-stable-quickjs 156 0 0
✅ nitro-stable-node 130 0 26
✅ nitro-stable-quickjs 130 0 26
✅ nuxt-stable-node 130 0 26
✅ nuxt-stable-quickjs 130 0 26
✅ sveltekit-stable-node 149 0 7
✅ sveltekit-stable-quickjs 149 0 7
✅ tanstack-start-node 130 0 26
✅ tanstack-start-quickjs 130 0 26
✅ vite-stable-node 130 0 26
✅ vite-stable-quickjs 130 0 26

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 156 0 0
✅ nextjs-turbopack-quickjs 156 0 0

✅ 🌐 Cross-language Conformance

App Passed Failed Skipped
✅ python 9 0 128

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit b85dbfc · Sat, 15 Aug 2026 00:35:42 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1376 (+73%) 🔻 1471 🔴 (+38%) 🔻 1496 🔴 (+35%) 🔻 1508 🔴 (-1.4%) 30
TTFS stream 1321 (+416%) 🔻 1438 🔴 (+33%) 🔻 1460 🔴 (+31%) 🔻 1502 🔴 (+9.4%) 30
TTFS hook + stream 1361 (+6.8%) 1752 🔴 (+24%) 🔻 1767 🔴 (+16%) 🔻 1843 🔴 (+10%) 30
Fan-out TTFS Promise.all(100 steps) 9110 (+2.3%) 10549 (±0%) 10601 (+0.6%) 15035 (+6.9%) 10
Fan-out TTLS Promise.all(100 steps) 17828 (-2.1%) 20350 (+2.6%) 24910 (+23%) 🔻 25018 (+4.2%) 10
STSO 1020 steps (inline) 119 (-0.8%) 304 (-29%) 💚 350 (-28%) 💚 524 (-25%) 💚 1019
WO 1020 steps 277603 (-25%) 💚 277603 (-25%) 💚 277603 (-25%) 💚 277603 (-25%) 💚 1
CRTT first chunk (pooled) 82 (-1.2%) 112 (-11%) 206 (+41%) 🔻 464 (+116%) 🔻 28

Streams

Scenario wr c/s rd c/s wr KiB/s rd KiB/s CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 100 (±0%) 101 (+2%) 5 (±0%) 5.1 (+2%) 91 (-7%) 114 (-17%) 226 (+31%) 559 (+71%) 80 (-33%) 10
size sweep (100/s, 160B-12KB) 100 (±0%) 101 (±0%) 334 (±0%) 335 (±0%) 104 (+2%) 129 (+4%) 306 (+79%) 839 (+167%) 122 (-5%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 89.2 (±0%) 89.2 (±0%) 16.2 (±0%) 16.2 (±0%) 135 (-1%) 114 (-24%) 156 (-47%) 579 (-31%) 275 (-55%) 3
replay eve-gpt-5.6-sol-2000t (1x) 54.7 (±0%) 54.7 (±0%) 355 (±0%) 355 (±0%) 284 (+137%) 107 (-14%) 152 (-9%) 455 (+34%) 529 (+21%) 2
replay eve-gpt-5.6-sol-2000t (2x) 109 (±0%) 109 (±0%) 710 (±0%) 710 (±0%) 110 (+4%) 154 (-6%) 235 (+7%) 617 (+59%) 246 (+1%) 3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 369121ms → this run 276301ms (Δ -92820ms, -25%)

  100-150 ms  █░░░┃                     main  16  this  62   +46
  150-200 ms  ████░░░░░░┃               main  49  this 139   +90
  200-250 ms  ████░░░░░░░░░░░░░░░░░░░┃  main  55  this 294  +239
  250-300 ms  █████████████░░░░░░░┃     main 161  this 261  +100
  300-350 ms  ████████████┃████         main 209  this 161   -48
  350-400 ms  ████┃██████████           main 186  this  56  -130
  400-450 ms  █┃███████████             main 160  this  21  -139
  450-500 ms  ┃███████                  main 103  this  11   -92
  500-550 ms  ┃███                      main  44  this   4   -40
  550-600 ms  ┃                         main  13  this   4    -9
  600-650 ms  ┃                         main   9  this   2    -7
  650-700 ms  ┃                         main   1  this   0    -1
  700-750 ms  ┃                         main   7  this   0    -7
  750-800 ms  ┃                         main   1  this   0    -1
  800-850 ms  ┃                         main   1  this   0    -1
  850-900 ms  ┃                         main   1  this   0    -1
  900-950 ms  ┃                         main   1  this   0    -1
1000-1050 ms  ┃                         main   1  this   1    +0
2100-2150 ms  ┃                         main   1  this   0    -1
3500-3550 ms  ┃                         main   0  this   1    +1
4300-4350 ms  ┃                         main   0  this   1    +1
4600-4650 ms  ┃                         main   0  this   1    +1
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant  RTT 1ms→5s+             avg         p50         p90          p99     n
control  ······█▆▁▁···   104.2 (-3%)    92 (-9%)  226 (+31%)   559 (+71%)  3000
sweep    ······██▂▁···  130.3 (+19%)   100 (-1%)  306 (+79%)  839 (+167%)  3000
gw 1x    ·····▁█▆▁▁···  107.7 (-22%)    96 (-9%)  156 (-47%)   579 (-31%)  5295
eve 1x   ·····▁█▄▁▁···   98.4 (-12%)   85 (-13%)   152 (-9%)   455 (+34%)  5186
eve 2x   ·····▁▇█▂▁···   126.6 (-8%)  108 (-13%)   235 (+7%)   617 (+59%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  ▇▅█▆▃▁▁▂▃▂  88–128ms
sweep    ▅▂▃▃▇█▅▂▁▂  102–174ms
gw 1x    █▃▂▃▂▁▂▂▁▁  90–178ms
eve 1x   ▄▂▄▂▁▂█▄▃▁  79–144ms
eve 2x   ▂▁▂▁▁▃▄█▂▂  101–212ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  █▇▅▃▁▁▂  128–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▁▇█▃▃▃▃▄▆▂  23–39ms
sweep    ▁▅▄▅█▃▅▃▃▄  30–64ms
gw 1x    █▃▃▃▁▂▄▂▁▂  25–43ms
eve 1x   ▃▂▄▂▁▂█▄▄▄  18–28ms
eve 2x   ▃▃▁▁▄▄█▆▂█  17–24ms
📜 Previous results (1)

6cd6cd2

Fri, 14 Aug 2026 23:57:45 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 340 (-57%) 💚 1446 🔴 (+35%) 🔻 1522 🔴 (+37%) 🔻 1722 🔴 (+13%) 30
TTFS stream 1301 (+408%) 🔻 1351 🔴 (+25%) 🔻 1368 🔴 (+23%) 🔻 1532 🔴 (+12%) 30
TTFS hook + stream 1485 (+17%) 🔻 1778 🔴 (+26%) 🔻 1804 🔴 (+19%) 🔻 2364 🔴 (+42%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 8906 (±0%) 10551 (±0%) 10792 (+2.4%) 14647 (+4.1%) 10
Fan-out TTLS Promise.all(100 steps) 17824 (-2.1%) 19630 (-1.0%) 21755 (+7.2%) 24709 (+2.9%) 10
STSO 1020 steps (inline) 117 (-2.5%) 360 (-15%) 💚 417 (-14%) 488 (-30%) 💚 1018
STSO 1020 steps (queue-hop) 2381 2381 2381 2381 1
WO 1020 steps 319803 (-13%) 319803 (-13%) 319803 (-13%) 319803 (-13%) 1
CRTT first chunk (pooled) 82 (-1.2%) 106 (-16%) 💚 122 (-16%) 💚 174 (-19%) 💚 28

Streams

Scenario wr c/s rd c/s wr KiB/s rd KiB/s CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 100 (±0%) 100 (+1%) 5 (±0%) 5 (±0%) 99.5 (+2%) 115 (-16%) 147 (-15%) 241 (-26%) 100 (-16%) 10
size sweep (100/s, 160B-12KB) 100 (±0%) 101 (+1%) 334 (±0%) 337 (+1%) 101 (-1%) 114 (-8%) 161 (-6%) 360 (+15%) 103 (-20%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 89.2 (±0%) 89.2 (±0%) 16.2 (±0%) 16.2 (±0%) 117 (-15%) 109 (-27%) 172 (-41%) 672 (-20%) 436 (-29%) 3
replay eve-gpt-5.6-sol-2000t (1x) 54.7 (±0%) 54.7 (±0%) 355 (±0%) 355 (±0%) 105 (-13%) 108 (-14%) 144 (-14%) 353 (+4%) 320 (-27%) 2
replay eve-gpt-5.6-sol-2000t (2x) 109 (±0%) 109 (±0%) 710 (±0%) 708 (±0%) 91 (-14%) 143 (-12%) 203 (-8%) 503 (+30%) 388 (+59%) 3
ℹ️ Metric definitions & methodology

Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). = main, = this run, = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

The `world` stub the reference snippets typecheck against declares only the
methods those snippets call, so the new `runs.waitForTerminalStatus()` sample
had no property to resolve. Declared optional, mirroring the World interface —
which is why the snippet reaches it through `?.`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
@vercel
vercel Bot temporarily deployed to Preview – workflow-docs August 14, 2026 23:35 Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves await run.returnValue latency by introducing an optional World-level long-poll (runs.waitForTerminalStatus) so callers can be notified as soon as a run becomes terminal instead of waiting for the next fixed ~1s polling tick.

Changes:

  • Add Storage['runs'].waitForTerminalStatus (optional) plus shared params type and documentation of its contract.
  • Update the core runtime polling loop to prefer waitForTerminalStatus (with pacing/kill switch) and fall back to interval runs.get.
  • Implement long-poll semantics across worlds (Vercel via /v2/runs/:id/status, Postgres via LISTEN/NOTIFY, Local via in-process signal + fs backstop) and add targeted tests.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
packages/world/src/runs.ts Adds WaitForTerminalRunStatusParams for the optional long-poll API.
packages/world/src/interfaces.ts Extends the Storage runs interface with optional waitForTerminalStatus and contract docs.
packages/world-vercel/src/storage.ts Wires runs.waitForTerminalStatus into the world-vercel storage implementation.
packages/world-vercel/src/runs.ts Implements /v2/runs/:id/status long-poll logic with budget clamping and degradation behavior.
packages/world-vercel/src/run-status-long-poll.test.ts Adds unit tests for world-vercel long-poll behavior and degradation cases.
packages/world-postgres/test/run-status-wait.test.ts Adds Postgres integration tests validating NOTIFY-driven wakeups and fallbacks.
packages/world-postgres/src/storage.ts Implements Postgres runs.waitForTerminalStatus and emits NOTIFY on terminal transitions.
packages/world-postgres/src/run-status.ts Adds shared LISTEN/NOTIFY listener + backstop interval configuration for terminal waits.
packages/world-postgres/src/index.ts Initializes and closes the shared run-status listener in the Postgres world lifecycle.
packages/world-local/src/storage/runs-storage.ts Implements local runs.waitForTerminalStatus with in-process signal and backstop poll.
packages/world-local/src/storage/run-status-wait.test.ts Adds fs-backed tests for local long-poll semantics (complete/cancel/timeout/abort).
packages/world-local/src/storage/run-status-signal.ts Adds in-process terminal-status signal emitter and backstop poll interval.
packages/world-local/src/storage/events-storage.ts Emits terminal signal after committing terminal run writes.
packages/core/src/runtime/run.ts Updates Run#pollReturnValue to prefer long-poll + pacing and adds env knobs/kill switch.
packages/core/src/runtime/run-return-value-long-poll.test.ts Adds fake-timer tests covering pacing, kill switch, and budget forwarding.
docs/content/worlds/v5/building-a-world.mdx Documents optional runs.waitForTerminalStatus and recommended implementation patterns.
docs/content/docs/v5/api-reference/workflow-runtime/world/storage.mdx Adds API reference docs for runs.waitForTerminalStatus and behavior notes.
.changeset/run-status-long-poll.md Changeset entry describing the new optional World long-poll and runtime behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/world-vercel/src/runs.ts Outdated
Comment on lines +275 to +281
const LONG_POLL_UNSUPPORTED_TTL_MS = 5 * 60 * 1000;

let longPollUnsupportedUntil = 0;

/** Test-only: forget that the long-poll route was unavailable. @internal */
export function _resetRunStatusLongPollSupportForTests(): void {
longPollUnsupportedUntil = 0;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in b1006b7. The suppression is now a Map<baseUrl, deadline> keyed off getHttpUrl(config).baseUrl, so a proxy that predates the route and a direct backend that has it no longer share a verdict. Covered by a new case (suppresses the fast path per backend, not process-wide) that 404s the direct host and then asserts the proxy is still long-polled.

Comment on lines +90 to +100
const ensureSubscribed = () => {
subscription ??= listenChannel(pool, RUN_STATUS_TOPIC, async (payload) => {
if (payload) emitter.emit(`run:${payload}`);
}).catch(() => {
// No listener connection available (pool options that don't permit a
// second client, a database without LISTEN). Waits degrade to the
// backstop re-read, which is the behavior of a plain poll.
return undefined;
});
return subscription;
};

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in b1006b7 — a failed attempt clears subscription so a later wait re-attempts.

One addition on top of the suggestion: a bare clear makes every waiting run retry the LISTEN once per poll interval while the database is unreachable, i.e. a connection attempt per second per awaited run against something already struggling. So the retry sits behind a 5s backoff (LISTEN_RETRY_BACKOFF_MS), which keeps a genuinely unavailable listener at one attempt every few seconds while still picking a restart back up well within a single 25s wait. close() pins the backoff at Infinity so a shut-down world can't re-open the connection.

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 Mint-ordered log — 3 fail of 41 total

log=mint-ordered · fence=per-spec

scenario outcome events virt replay violations
smoke-no-steps completed 3 0ms ok 0
smoke-one-step completed 6 0ms ok 0
hook-at-step-started completed 12 0ms ok 0
hook-at-step-completed completed 12 0ms ok 0
hook-at-hook-created completed 12 0ms ok 0
deadline-hook-wins completed 7 1.0h ok 0
deadline-expires completed 7 1.0h ok 0
long-sleep completed 11 30.0d ok 0
hook-never-arrives stalled 3 0ms skipped 0
step-retries-twice completed 10 2.0s ok 0
parallel-steps completed 9 0ms ok 0
hook-on-execution-state completed 12 0ms ok 0
peek-hook-before-branch completed 12 0ms ok 0
peek-hook-after-branch completed 12 0ms ok 0
peek-hook-at-registration completed 12 0ms ok 0
race-hook-before-probe completed 12 0ms ok 0
race-hook-after-probe completed 12 0ms ok 0
race-duplicate-delivery completed 13 0ms ok 0
attr-hook-before-step completed 11 0ms ok 0
attr-hook-after-step completed 11 0ms ok 0
attr-from-step-body completed 13 0ms ok 0
fork-hook-after-timeout completed 14 1.0m ok 0
fork-hook-before-timeout completed 14 1.0m ok 0
count-hook-after-timeout completed 17 1.0m ok 0
count-hook-before-timeout completed 20 1.0m ok 0
stale-read-step-count-fork completed 20 1.0m ok 0
stale-read-equal-step-counts completed 14 1.0m ok 0
step-vs-step-fork completed 12 0ms ok 0
step-vs-step-fork-fenced completed 12 0ms ok 0
fence-catches-benign-direction completed 12 5ms ok 0
in-flight-before-decision failed 9 1.0m MISMATCH 1
in-flight-before-decision-counted failed 9 1.0m MISMATCH 1
in-flight-after-decision failed 9 1.0m MISMATCH 1
stale-read-step-count-fork-fenced completed 20 1.0m ok 0
fork-hook-wins completed 13 1.0m ok 0
fork-timeout-wins completed 13 1.0m ok 0
unclaimed-payload-under-fork completed 17 1.0m ok 0
claimed-payload-under-fork completed 17 1.0m ok 0
writers-independent-step-bodies completed 12 0ms ok 0
writers-scripted-tempo completed 12 0ms ok 0
cancel-mid-step cancelled 7 0ms skipped 0

Full trace: world-sim-mint.txt

🟢 Append-only log — 0 fail of 41 total

log=append-only · fence=per-spec

scenario outcome events virt replay violations
smoke-no-steps completed 3 0ms ok 0
smoke-one-step completed 6 0ms ok 0
hook-at-step-started completed 12 0ms ok 0
hook-at-step-completed completed 12 0ms ok 0
hook-at-hook-created completed 12 0ms ok 0
deadline-hook-wins completed 7 1.0h ok 0
deadline-expires completed 7 1.0h ok 0
long-sleep completed 11 30.0d ok 0
hook-never-arrives stalled 3 0ms skipped 0
step-retries-twice completed 10 2.0s ok 0
parallel-steps completed 9 0ms ok 0
hook-on-execution-state completed 12 0ms ok 0
peek-hook-before-branch completed 12 0ms ok 0
peek-hook-after-branch completed 12 0ms ok 0
peek-hook-at-registration completed 12 0ms ok 0
race-hook-before-probe completed 12 0ms ok 0
race-hook-after-probe completed 12 0ms ok 0
race-duplicate-delivery completed 13 0ms ok 0
attr-hook-before-step completed 11 0ms ok 0
attr-hook-after-step completed 11 0ms ok 0
attr-from-step-body completed 13 0ms ok 0
fork-hook-after-timeout completed 14 1.0m ok 0
fork-hook-before-timeout completed 14 1.0m ok 0
count-hook-after-timeout completed 17 1.0m ok 0
count-hook-before-timeout completed 20 1.0m ok 0
stale-read-step-count-fork completed 20 1.0m ok 0
stale-read-equal-step-counts completed 14 1.0m ok 0
step-vs-step-fork completed 12 0ms ok 0
step-vs-step-fork-fenced completed 12 0ms ok 0
fence-catches-benign-direction completed 12 5ms ok 0
in-flight-before-decision completed 17 1.0m ok 0
in-flight-before-decision-counted completed 17 1.0m ok 0
in-flight-after-decision completed 19 2.0m ok 0
stale-read-step-count-fork-fenced completed 20 1.0m ok 0
fork-hook-wins completed 13 1.0m ok 0
fork-timeout-wins completed 13 1.0m ok 0
unclaimed-payload-under-fork completed 17 1.0m ok 0
claimed-payload-under-fork completed 17 1.0m ok 0
writers-independent-step-bodies completed 12 0ms ok 0
writers-scripted-tempo completed 12 0ms ok 0
cancel-mid-step cancelled 7 0ms skipped 0

Full trace: world-sim-append-only.txt

Copy link
Copy Markdown
Contributor Author

CI state

Green except two E2E Local Dev Tests cells (nextjs-turbopack - stable node, nextjs-webpack - canary node) and the E2E Required Check that aggregates them.

Both fail on the same test — packages/core/e2e/dev.test.ts > dev e2e > should follow Next flow-route HMR rebuild rules for body-only changes — waiting on Next dev-server HMR rebuild counts / manifest propagation (expected 5 to be 1, expected 3 to be 1, and a 300s manifest timeout). It is failing on main independently of this branch: run 31850646225 → E2E Local Dev Tests (nextjs-webpack - stable node) hit the identical assertion (expected 2 to be 1) ~12 minutes before this PR's run, in a different matrix cell. Nothing in this PR touches dev mode, HMR, or manifest generation.

Worth calling out what did pass: all 27 E2E Vercel Prod Tests cells are green, and they run world-vercel against the production workflow-server — which does not have GET /v2/runs/:runId/status yet. So the degradation path in this PR (long poll 404s → plain read → mark unsupported for 5 minutes) is exercised end-to-end on every awaited run in that matrix, and await run.returnValue behaves exactly as it does today against a server that predates the route.

Two review findings on the degradation paths:

world-vercel cached "this backend has no /status route" process-wide, but one
process can hold worlds pointed at different backends — the api.vercel.com
proxy and workflow-server directly resolve to different hosts, which can be on
different versions. Key the suppression by resolved base URL so a miss against
one never disables the fast path for the other.

world-postgres cached a *failed* LISTEN attempt forever, so a database restart
or a brief blip at process start would degrade every wait to backstop polling
for the life of the process. Clear it on failure and re-attempt, behind a 5s
backoff so an unavailable listener costs one connection attempt every few
seconds rather than one per waiting run per poll interval. close() pins the
backoff open so a shut-down world never re-opens it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants