Repository navigation
Templates aren't exercised by CI: 5 chat templates were broken for days #2488
Description
The immediate problem (fixed in #2486)
shinychat 0.7.0 (2026-09-04) made Chat(messages=...) a hard error unless history=False is also passed:
ValueError: `Chat(messages=...)` requires `history=False`: startup messages
can't be recorded by the conversation-history feature. Use the `greeting`
parameter for a startup message, ...
Five shiny/templates/chat/** apps passed messages= to the constructor, so they raised on startup — shiny create produced an app that wouldn't run:
templates/chat/llm-enterprise/azure-openai/app.pytemplates/chat/llms/langchain/app.pytemplates/chat/llms/anthropic/app.pytemplates/chat/llms/ollama/app.pytemplates/chat/llms/openai/app.py
Those five are fixed in #2486 (messages= → greeting=).
The actual issue
Nothing in CI runs the templates. tests/playwright/examples covers api-examples/ and got caught immediately — 30 red jobs. The templates were equally broken and completely silent; they were only found by grepping for the same deprecated call while fixing the examples.
So the question isn't these five files, it's: how would we ever notice next time? A template that doesn't start is arguably worse than a failing test, since it's the first thing a new user runs.
Worth considering:
- A smoke test that instantiates each template's app object — most of these failures are import/construction-time, so it needn't drive a browser to catch them.
- The LLM templates need API keys to do anything real, but constructing the app shouldn't. Worth checking how far a no-key run gets.
- A periodic (not per-PR) job may be the right tradeoff if a full pass is slow.
Related
Whatever shape this takes should also cover the deprecation surface more broadly — the bundled shiny-for-python Agent Skill was teaching messages= in four places (also fixed in #2486), which is another artifact nothing validates.
Two playwright-shiny tests are still red on main from the same shinychat change and need a chat-owner decision, tracked in #2486's description: bookmark/chat/client_state (uses messages= on the constructor, but its test asserts expect_messages, and greeting= renders outside loc_messages) and components/chat/icon (asserts the assistant icon on a startup message that greeting= doesn't create).
Activity
pm25coder commentedon Sep 7, 2026
The "how would we ever notice next time?" question has a sharper version: the five files and
tests/playwright/examplesare the same codebase at the same moment, and only one side was loud — the divider is which artifact CI instantiates, not which code changed.api-examples/got 30 red jobs because something constructs it;templates/was silent because nothing does. A new-user artifact is the worst place for that silence, exactly as you say.Three shapes worth considering, in order of cost:
Derive the artifact set by directory scan, not by adding the five. A smoke job that walks
shiny/templates/**, and for each template runs the same scaffold a new user runs (shiny create→ import the app module → construct the app object), fails by construction when a template is added — the check list and the artifact tree can't drift apart. The observed failure class is import/construction-time (Chat(messages=...)raised at construction, no browser, no keys), so a no-key construction pass should cover it; the "how far does a no-key run get" question is worth one experiment, and for the LLM templates the answer is usually "further than you'd think", because clients are typically constructed at request time, not at app import.Make the discovery mechanism a permanent check. These were found by grepping for the deprecated call while fixing something else — that grep is itself the cheapest regression guard. A lint rule that bans the deprecated constructor parameter inside
templates/andexamples/(or anywhere undershiny/) fails at PR time with a precise message, instead of days later at a user's firstshiny create. Deprecations that became hard errors are exactly the class a text-level rule catches, and it costs nothing at runtime.Guard the "covered" list against new holes. The subtle failure is not these five — it's that a list of checked examples reads as "templates are covered" while whole directories sit outside it. Whatever lands, the durable property to assert is: every entry under
shiny/templates/is instantiable, enumerated from the filesystem. That converts "remember to add your template to the test" into something a new template cannot skip silently.Worth noting the construction-time smoke test and the grep rule catch different halves: the smoke test proves the artifact runs, the rule proves the migration can't quietly regress. Both are cheap enough to run per-PR, which matters here because the daily scheduled run is exactly how long "for days" gets to be.