feat: Run reactive effects

.claude/references/architecture.md

@@ -15,8 +15,10 @@ The reactive system is based on a **push-pull** model with three core abstractio
   sources (Value) it reads from.
 - **Dependents**: Each reactive source maintains a list of downstream consumers
   that depend on it. When a source invalidates, it notifies all dependents.
-- **ReactiveEnvironment**: Global singleton managing the reactive graph,
-  execution queue, and flush cycles.
+- **ReactiveEnvironment**: Global singleton managing the reactive graph, the
+  reactive effect queue, and rounds. Its class docstring in
+  `shiny/reactive/_core.py` defines the terms used for the reactive system:
+  reactive effect queue, round, idle, cycle, and output flush.
 
 Key implementation details:
 
@@ -25,8 +27,9 @@ Key implementation details:
 - `Effect_()` is a side-effect that re-executes when dependencies change
 - `event()` decorator suppresses reactive dependencies for specific reads
 - The reactive graph is built automatically through the Context's dependency tracking
-- Execution uses a priority queue to ensure correct invalidation ordering
-- In tests, `reactive.flush()` forces a synchronous flush of the reactive graph
+- Each round takes effects from the reactive effect queue in priority order
+- In tests, `await reactive.flush()` runs rounds until the reactive environment is
+  idle
 
 ## Session Hierarchy
 

CHANGELOG.md

@@ -7,8 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Breaking changes
+
+* OpenTelemetry `reactive_update` spans now follow each session's cycle: one span per session, from when the session turns busy until all of its effects (including async ones) have finished, carrying the session's `session.id`. The session's calc, effect, and output spans are its children. Before, there was one span per reactive flush, which can serve several sessions, so with concurrent effects it couldn't carry a single `session.id` and ended before async effects finished. Effects without a session no longer have a `reactive_update` parent. (#2522)
+
+* Reactive effects now run concurrently. Shiny now starts each invalidated effect without waiting for an `async` effect's awaited part, so a slow `async` effect, calc, or render function no longer delays other sessions. Within a session, input changes and `reactive.invalidate_later()` still wait until all of that session's effects have finished, and outputs are sent together once they have. What app authors may notice:
+
+    * `async` effects interleave at each `await` instead of running one after another. `priority` orders when effects start, not when they finish. Runs of the same effect still don't overlap: a re-run waits for the previous run.
+
+    * `reactive.lock()` no longer pauses reactive processing: Shiny itself no longer takes it (see Deprecations). To change reactive state from another `asyncio` task, set the value directly; its dependents re-run automatically, so there's no need to `await reactive.flush()`.
+
+    * `await reactive.flush()` returns right away, without waiting for dependents, when called from within an effect, a `reactive.on_flushed()` or `session.on_flush()`/`on_flushed()` callback, or a task one of these started while it is still running, since waiting there could deadlock. A task that outlives the effect that started it, such as an extended task's body, still waits. The wait covers the whole process, including other sessions' effects and output.
+
+    * `reactive.on_flushed()` callbacks run after each round of effects has *started*, without waiting for `async` effects, and can run several times per `reactive.flush()`. Before, they ran once all effects had finished. An error in one of these callbacks is now printed, and the other callbacks still run, instead of closing a session.
+
+    * Message handlers (`session.set_message_handler()`) run in their own task, so a slow handler no longer holds up other messages from the client.
+
+    * Each session sends its outputs in a task of its own, once all of its effects have finished. A slow client, or a slow `session.on_flush()` callback, delays only its own session's output, not other sessions' output or other reactive work.
+
+    * Setting a `reactive.value` now re-runs its dependents automatically, without a call to `reactive.flush()`. The reactive graph itself still isn't thread-safe, so from another thread use `loop.call_soon_threadsafe(value.set, new_value)`.
+
+    * File uploads and `session.dynamic_route()` requests no longer mark the session busy, matching Shiny for R.
+
+    * Destroying an effect (`effect.destroy()`) cancels its in-progress run, raising `asyncio.CancelledError` at its current `await`. So ending a session, or `session.destroy(id)`, cancels its in-progress `async` effects and render functions instead of letting them run on after the session or module is gone.
+
+  (#2508, #2515)
+
 ### New features
 
+* Added `session.run_once_when_idle(fn)`, which runs a function once, at the start of the session's next cycle: once all of the session's effects have finished and its outputs have been sent. Input changes from the client and `reactive.invalidate_later()` already wait this way, so effects that are still running keep seeing the same values; use it to change reactive state the same way from a background task or a custom timer. `reactive.invalidate_later()` now uses it instead of a private method. (#2517)
+
 * `ui.sidebar()` gains a `role` parameter (`"form"`, `"search"`, `"complementary"`, or `"region"`) for opt-in ARIA landmark markup (rstudio/bslib#1359). `"complementary"` renders an `<aside>`, other roles render a `<div>` with the corresponding `role` attribute, and landmark roles require an accessible name (from `title`, `aria_label`, or `aria_labelledby`). Note that the default (`role=None`) now renders a neutral `<div>` instead of an `<aside>`. In addition, `ui.page_sidebar()` now places the whole sidebar layout inside the page's `<main>` landmark. (#2526)
 
 ### Improvements
@@ -19,8 +47,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 * In Express, `@output_args()` now accepts every argument of the output's UI function, with matching types. `@render.ui` now takes `inline`, `container`, `fill`, `fillable`, and HTML attributes, instead of raising a `TypeError`. `@render.text` now takes `container`. `@render.plot` and `@render.image` now declare `inline`, `click`, `dblclick`, `hover`, `brush`, and `fill` explicitly (and `width` and `height` for `@render.image`), instead of accepting any keyword. `@render.download_button()` and `@render.download_link()` now take `label`. (#2539)
 
+### Deprecations
+
+* `reactive.lock()` is deprecated and now emits a `ShinyDeprecationWarning` (once per process). It does nothing else: it returns an `asyncio.Lock` that never blocks, so code that holds it no longer keeps out other code that holds it. Code that relies on it for mutual exclusion should create its own `asyncio.Lock`. Remove `async with reactive.lock():` and the `await reactive.flush()` that usually follows it, and set the reactive value directly. To apply a change only once a session's running effects have finished, use `session.run_once_when_idle()`. (#2520)
+
 ### Bug fixes
 
+* UI updates and custom messages sent outside an output or effect, such as `ui.update_text()` in a `session.on_flushed()` callback or in an `asyncio` task, or `ui.insert_ui()` from a message handler, are now sent to the client right away. Before, they waited until the next input change. (#1889, #21)
+
+* Setting a reactive value from a download handler (`@render.download_button`) now updates the outputs and effects that read it right away, including while a streamed download is still sending. Before, they updated only after the next message from the client. (#1785)
+
 * `Jsonifiable` now uses `Sequence` and `Mapping` for its container arms instead of `List` and `Dict`, which are invariant. Values like `dict[str, int]` and `list[str]` now type-check as `Jsonifiable`, including as the return type of a function passed to a `Renderer[Jsonifiable]`. As a side effect, type checkers also accept other sequences and mappings, such as `range`, `deque`, and `bytes`, even though only `list`, `tuple`, and `dict` serialize. Nothing changes at runtime. (Thanks, @wolfgang-aura!) (#2511)
 
 * `Progress.set(0)` now places the bar at `0` relative to `min` and `max`, instead of sending `0` unnormalized. With a negative `min`, such as `Progress(min=-10, max=10)`, a value of `0` showed an empty bar rather than a half-full one. (#2518)

CLAUDE.md

@@ -192,7 +192,7 @@ CI then rejects.
 - **Missing HTML deps**: If styles don't work, ensure `components_dependencies()` is included
 - **Express mode confusion**: Remember Express uses AST transformation; debugging requires understanding the transformed code
 - **Session context errors**: Always access `input`/`output` within reactive context
-- **Reactive graph debugging**: Use `reactive.flush()` to force synchronous execution in tests
+- **Reactive graph debugging**: `await reactive.flush()` runs rounds until the reactive environment is idle (tests)
 - **Playwright timing**: Use `.expect_*()` methods which auto-wait; avoid manual `sleep()`
 - **Asset updates**: After running `make upgrade-html-deps`, verify theme preset files were updated
 - **Out-of-sync Playwright controllers**: When adding or altering UI components, always create or update their companion Playwright controller in `shiny/playwright/controller/` with matching locators, action methods, and expectation methods, and export them from `shiny/playwright/controller/__init__.py`

docs/_quartodoc-core.yml

@@ -237,7 +237,6 @@ quartodoc:
         - reactive.flush
         - reactive.poll
         - reactive.file_reader
-        - reactive.lock
         - req
     - title: Create and run applications
       desc: ""
@@ -296,6 +295,7 @@ quartodoc:
             - session.Session.on_flush
             - session.Session.on_flushed
             - session.Session.on_ended
+            - session.Session.run_once_when_idle
             - session.Session.dynamic_route
             - session.Session.allow_reconnect
             - session.Session.close
@@ -414,6 +414,7 @@ quartodoc:
         - render.download
         - render.transformer.output_transformer
         - render.transformer.resolve_value_fn
+        - reactive.lock
         # `ui.update_navs()` is deprecated (superseded by `ui.update_navset()`) but
         # still exists. It is deliberately left undocumented; uncomment to publish it.
         # - ui.update_navs

docs/_quartodoc-express.yml

@@ -134,7 +134,6 @@ quartodoc:
         - reactive.flush
         - reactive.poll
         - reactive.file_reader
-        - reactive.lock
         - req
     - title: Reusable Express code
       desc: Create reusable Express code.
@@ -255,3 +254,4 @@ quartodoc:
       desc: ""
       contents:
         - express.render.download
+        - reactive.lock

pytest.ini

@@ -5,6 +5,14 @@ asyncio_default_fixture_loop_scope=function
 testpaths=tests/pytest/
 ; Note: Browsers are set within `./Makefile`
 addopts = --strict-markers --durations=6 --durations-min=5.0 --numprocesses auto
+; No test may take more than 2 minutes, so a hang fails instead of running until
+; CI cancels the job. The `thread` method kills the hung xdist worker (which
+; xdist reports as a failure of that test); its own stack dump is lost in the
+; worker's IPC pipe, so faulthandler dumps every thread's stack to stderr first.
+; (Same approach as the Playwright targets in the Makefile.)
+timeout = 120
+timeout_method = thread
+faulthandler_timeout = 110
 ; Tests must not emit stray warnings; assert expected ones with `pytest.warns()`.
 ; Third-party deprecations we can't act on go in the ignore list below.
 filterwarnings =
@@ -24,4 +32,7 @@ filterwarnings =
     ; test at fault -- i.e. they make the suite flaky instead of strict.
     ignore::ResourceWarning
     ignore::pytest.PytestUnraisableExceptionWarning
+    ; When a timeout kills an xdist worker, syrupy warns that the killed worker's
+    ; tests are missing; as an error, that hides the summary naming the hung test.
+    ignore:.*selected test\(s\) missing from collected items:UserWarning
 ; verbosity_test_cases=2

shiny/.agents/skills/shiny-for-python/SKILL.md

@@ -20,7 +20,7 @@ reference file before writing code** for that area.
 
 | Topic | Use when | Reference |
 |---|---|---|
-| Reactivity | A value should recompute or an output update as inputs change; choosing between calc / effect / value; `req`, `isolate`, timers, polling | `references/reactivity.md` |
+| Reactivity | A value should recompute or an output update as inputs change; choosing between calc / effect / value; `req`, `isolate`, timers, polling, setting values from background tasks | `references/reactivity.md` |
 | Express mode | Writing or converting an Express app (`from shiny.express import ...`); context-manager layout; `page_opts`, `@expressify` | `references/express.md` |
 | Modules (Core) | A reusable, repeatable UI+server component in a Core app; avoiding input/output id collisions across copies | `references/modules-core.md` |
 | Modules (Express) | The same reusable-component need in an Express app, via the single `@module` decorator | `references/modules-express.md` |
@@ -57,7 +57,7 @@ reference file before writing code** for that area.
 | Topic | Use when | Reference |
 |---|---|---|
 | Feedback | Toasts/notifications, modal dialogs, progress bars, busy indicators | `references/feedback.md` |
-| Extended tasks | Running slow work off the reactive flush without freezing the app; task buttons | `references/extended-tasks.md` |
+| Extended tasks | Running slow work outside the session's reactive cycle so the session stays responsive; task buttons | `references/extended-tasks.md` |
 | Bookmarking | Saving/restoring app state — shareable URLs, refresh persistence, server-side state | `references/bookmarking.md` |
 
 ## AI & streaming

shiny/.agents/skills/shiny-for-python/references/extended-tasks.md

@@ -3,15 +3,18 @@
 ## Overview
 
 A slow computation inside a `@reactive.calc`, `@reactive.effect`, or `@render.*`
-blocks the reactive flush - even an `async def` one - so the whole session (and,
-for shared resources, other sessions) freezes until it returns. Do NOT put a
-slow API call, model inference, or long loop directly in a calc/effect.
+keeps its session busy until it returns: that session applies no input changes
+and sends no outputs in the meantime. A sync one also blocks every session in the
+process, since it holds the event loop. (Other sessions keep running while an
+`async def` one awaits.) Do NOT put a slow API call, model inference, or long loop
+directly in a calc/effect.
 
 `@reactive.extended_task` runs an **async** function in a background asyncio
-task, off the reactive flush. Reactivity keeps processing while it runs; the
-result flows back into the reactive graph when it finishes.
+task, outside the session's reactive cycle. The session keeps processing inputs
+and outputs while it runs; the result flows back into the reactive graph when it
+finishes.
 
-Because it runs outside the reactive flush, the task **cannot read reactive
+Because it runs outside the reactive graph, the task **cannot read reactive
 sources** (`input.x()`, a `reactive.value`, a calc). Read those values *before*
 invoking and pass them in as arguments; reading one inside raises an error.
 
@@ -121,6 +124,9 @@ current one finishes.
 - Long synchronous work (a blocking library call) inside the async task still
   blocks the event loop - offload it with `asyncio.to_thread(...)` or an async
   client.
+- Calling `value.set(...)` from that worker thread -> the reactive graph isn't
+  thread-safe and it can raise. Return the result to the task instead, or hand the
+  call to the event loop: `loop.call_soon_threadsafe(value.set, new_value)`.
 - Calling `task.result()` outside a reactive context -> no dependency is tracked
   and it errors; read it inside a `@render.*`, calc, or effect.
 - Expecting a fresh `.invoke()` to interrupt the running task -> it queues

shiny/.agents/skills/shiny-for-python/references/otel.md

@@ -39,11 +39,18 @@ At the `all` level the span hierarchy looks like:
 
 ```text
 session_start
-  └─ reactive_update
-      ├─ reactive.calc filtered_data
+  └─ reactive_update              # the session's first cycle
       └─ output result
+          └─ reactive.calc filtered_data
+reactive_update                   # each later cycle: a trace of its own
+  └─ output result
+      └─ reactive.calc filtered_data
 ```
 
+A `reactive_update` span covers one session cycle: from when the session turns
+busy until all of its effects, including async ones, have finished. It carries
+`session.id`. Effects without a session have no `reactive_update` parent.
+
 ## Collection levels: `SHINY_OTEL_COLLECT`
 
 Controls how much Shiny telemetry is emitted (default `all`):
@@ -52,7 +59,7 @@ Controls how much Shiny telemetry is emitted (default `all`):
 |---|---|---|
 | `none` | nothing from Shiny | disable Shiny spans, keep your own |
 | `session` | session lifecycle only | low-overhead production |
-| `reactive_update` | + one span per flush cycle | balanced production |
+| `reactive_update` | + one span per session cycle | balanced production |
 | `reactivity` | + per calc/effect/output spans, value-update logs | development, debugging |
 | `all` | everything (currently = `reactivity`) | maximum detail |
 

shiny/.agents/skills/shiny-for-python/references/reactivity.md

@@ -144,6 +144,47 @@ def data2():
 Declare `poll`/`file_reader` at module top level to share one cache across
 sessions.
 
+To change reactive state from a background `asyncio` task, call `.set()`
+directly. Setting a value schedules a flush, so there's no lock to take and no
+`reactive.flush()` to await. (From another thread, use
+`loop.call_soon_threadsafe(v.set, x)`.) For a producer shared by all sessions
+(e.g. one polling an API), define it at module top level and start it from the
+first session, once the event loop is running. Keep a reference to its task,
+since the event loop holds tasks only weakly:
+
+```python
+import asyncio
+import contextvars
+import logging
+
+from shiny import reactive
+
+latest = reactive.value(None)
+_tasks: set[asyncio.Task[None]] = set()
+
+async def _produce():
+    while True:
+        try:
+            latest.set(await fetch())  # your async data source
+        except Exception:
+            logging.exception("fetch failed; retrying")  # keep the producer alive
+        await asyncio.sleep(10)
+
+def server(input, output, session):
+    if not _tasks:
+        # A fresh context keeps this session's context out of the shared task.
+        _tasks.add(contextvars.Context().run(asyncio.create_task, _produce()))
+```
+
+A session's effects that are paused at an `await` see a direct `.set()` right
+away. To apply the change the way input changes and `invalidate_later` do, hand
+it to the session. It runs once, after the session's running effects finish, so
+they keep seeing stable values:
+
+```python
+session.run_once_when_idle(lambda: latest.set(new_value))
+```
+
 (For long-running async work that must not block the session, see
 `reactive.extended_task` - out of scope here.)
 
@@ -158,8 +199,9 @@ sessions.
 | Wait for / validate a value | `req(x)`, `req(x, cancel_output=True)` |
 | Read without depending | `with reactive.isolate():` |
 | Timer | `reactive.invalidate_later(secs)` |
+| Set a value from a background task | `v.set(x)`; between cycles: `session.run_once_when_idle(fn)` |
 | Poll a data source / file | `@reactive.poll(...)` / `@reactive.file_reader(...)` |
-| Force sync execution in tests | `reactive.flush()` |
+| Run reactive work until idle (tests) | `await reactive.flush()` |
 
 ## Common mistakes
 
@@ -176,3 +218,7 @@ sessions.
   `isolate()` -> self-invalidating loop; wrap the read in `reactive.isolate()`.
 - `while`/`sleep` loop to watch a DB or file -> use `reactive.poll` or
   `reactive.file_reader`.
+- `async with reactive.lock():` around a `.set()`, followed by
+  `await reactive.flush()` -> `reactive.lock()` is deprecated and does nothing,
+  and the flush adds nothing but a wait on every session's effects. Call
+  `.set()` alone.