Repository navigation
Add a refresh-cache flag (--no-cache with --cache-registry) for manual workflow dispatches #847
Description
Background
Part of the eMudhra intermediate CA compromise response (#849). Rebuilt images must pick up the latest distro ca-certificates. Registry layer caching stops that from happening. The apt-get update && apt-get upgrade && … install ca-certificates RUN layer gets its cache key from the parent layer and the command text, so unless the upstream base image digest changed, BuildKit reuses the old layer and the upgrade never runs.
Evidence from images-connect production runs:
| Run | Event | apt-upgrade layers run | apt-upgrade layers reused from cache |
|---|---|---|---|
| 39478407535 (Oct 20) | push | 0 | 36 / 36 |
| 39090338821 (Oct 18) | schedule | 21 | 15 |
Why the existing cache: false input isn't enough
In bakery-build-native.yml, cache: false only drops --cache-registry. That turns off cache reads and writes. The rebuild would publish patched images, but the registry cache tags (e.g. ghcr.io/posit-dev/connect/cache:2026.10.0-ubuntu-24.04-std-amd64) would still hold the old upgrade layer. The next push or scheduled build would reuse that layer and overwrite the patched tags with images that trust the compromised intermediates again.
Approach
Run the build with bakery build --no-cache and --cache-registry. BuildKit then ignores the cache on import, rebuilds every layer, and still writes the fresh layers to the cache registry (bakery adds cache-to with mode=max whenever --push and --cache-registry are both set). Later cached builds then reuse the patched layers.
Tasks
Upstream (images-shared)
- Add a boolean input to
bakery-build-native.yml, e.g.refresh-cache(defaultfalse): "Rebuild all layers without reading the build cache, but still write fresh layers to the cache registry." The existingcacheinput means "use the cache registry at all", so giving the new input a separate name avoids an ambiguouscache/no-cachepair. - When
refresh-cacheistrue, add--no-cacheto thebakery buildcall in thebuild-testjob and leaveCACHE_FLAGS(--cache-registry) as is. - Test it on one image: run the build with
refresh-cache: trueand check that the logs show noCACHEDsteps, that the cache tag in GHCR has a new digest/timestamp, and that a follow-up normal build shows the apt-upgrade step asCACHEDfrom the new cache entry (the installedca-certificatesversion should match the patched release). - Update the
v0tag so downstream@v0callers pick up the change. - Optional:
bakery-build.ymlhas no callers in the product repos. Mirror the input there for consistency if it's cheap. Its push step doesn't pass--cache-registry, so it never writes the registry cache anyway.
Downstream (blocked on the upstream change and the v0 tag update)
- Add a
refresh-cachebooleanworkflow_dispatchinput (defaultfalse) and passrefresh-cache: ${{ inputs.refresh-cache || false }}to the shared workflow (inputs.*is null for non-dispatch events;|| falsekeepsschedule/pushunchanged):images-connect:production.yml,content.ymlimages-package-manager:production.ymlimages-workbench:production.yml,session.yml(bothuses:calls)- Optional:
development*.ymlin each repo, for consistency
-
images-specialized: these workflows callbakery builddirectly, not the shared workflow.wgcw.yml: add a dispatch input that adds--no-cacheto the--loadbuild step. Leave the later--pushstep without--no-cacheso it reuses the locally built, tested layers. Note that this workflow passes--cache-registryonly to the--loadstep, so it reads the registry cache but never writes it (bakery only addscache-towith--push). Find out what refreshes that cache. If nothing does,wgcwbuilds will keep reading stale layers after this rebuild.positron-sagemaker.yml,rstudio-sagemaker.yml,waml.yml: no registry cache and GitHub-hosted runners start fresh, so a plain dispatch already rebuilds clean. No change needed.
Acceptance criteria
- A manual dispatch with
refresh-cache: truerebuilds every layer, installs the current distroca-certificates, and refreshes the registry cache entries. - Later scheduled and push builds reuse the refreshed cache and don't bring back the old
ca-certificates. - Scheduled, push, and PR builds behave as before when the flag isn't set.
Activity
[-]Add a flag to disable build cache in manual workflow dispatches[/-][+]Add a refresh-cache flag (--no-cache with --cache-registry) for manual workflow dispatches[/+]ianpittwood commentedon Oct 21, 2026
Downstream PRs, opened as drafts until #850 merges and
v0is updated:production.yml,content.ymlproduction.ymlproduction.yml,session.yml(both calls)wgcw.yml(--no-cacheon the--loadstep only)Remaining for #847: merge #850, update
v0, mark these ready, and merge them.ci: add refresh-cache dispatch input to rebuild without reading build…
ianpittwood commentedon Oct 23, 2026
Done.
refresh-cache: truerunsbakery build --no-cachewhile keeping--cache-registry, so nothing is read from the build cache and fresh layers are written back. This was verified against a local registry and on the real GHCR cache during the #848 rebuilds.Shared workflow:
refresh-cacheinput inbakery-build-native.yml/bakery-build.yml, released tov0Dispatch input in the product repos:
production.yml,content.yml)production.yml)production.yml,session.yml)wgcw.yml; effectively a no-op, since that cache never worked; removal tracked in Remove the non-functional registry cache from WGCW builds (treat like WAML) images-specialized#122)Related: the
recentdispatch input and its0-parsing fix are attached to #851.