Skip to content

Add a refresh-cache flag (--no-cache with --cache-registry) for manual workflow dispatches #847

Closed

Description

@ianpittwood

Background

Part of the eMudhra intermediate CA compromise response (#849). Rebuilt images must pick up the latest distro ca-certificates. Registry layer caching stops that from happening. The apt-get update && apt-get upgrade && … install ca-certificates RUN layer gets its cache key from the parent layer and the command text, so unless the upstream base image digest changed, BuildKit reuses the old layer and the upgrade never runs.

Evidence from images-connect production runs:

Run Event apt-upgrade layers run apt-upgrade layers reused from cache
39478407535 (Oct 20) push 0 36 / 36
39090338821 (Oct 18) schedule 21 15

Why the existing cache: false input isn't enough

In bakery-build-native.yml, cache: false only drops --cache-registry. That turns off cache reads and writes. The rebuild would publish patched images, but the registry cache tags (e.g. ghcr.io/posit-dev/connect/cache:2026.10.0-ubuntu-24.04-std-amd64) would still hold the old upgrade layer. The next push or scheduled build would reuse that layer and overwrite the patched tags with images that trust the compromised intermediates again.

Approach

Run the build with bakery build --no-cache and --cache-registry. BuildKit then ignores the cache on import, rebuilds every layer, and still writes the fresh layers to the cache registry (bakery adds cache-to with mode=max whenever --push and --cache-registry are both set). Later cached builds then reuse the patched layers.

Tasks

Upstream (images-shared)

  • Add a boolean input to bakery-build-native.yml, e.g. refresh-cache (default false): "Rebuild all layers without reading the build cache, but still write fresh layers to the cache registry." The existing cache input means "use the cache registry at all", so giving the new input a separate name avoids an ambiguous cache / no-cache pair.
  • When refresh-cache is true, add --no-cache to the bakery build call in the build-test job and leave CACHE_FLAGS (--cache-registry) as is.
  • Test it on one image: run the build with refresh-cache: true and check that the logs show no CACHED steps, that the cache tag in GHCR has a new digest/timestamp, and that a follow-up normal build shows the apt-upgrade step as CACHED from the new cache entry (the installed ca-certificates version should match the patched release).
  • Update the v0 tag so downstream @v0 callers pick up the change.
  • Optional: bakery-build.yml has no callers in the product repos. Mirror the input there for consistency if it's cheap. Its push step doesn't pass --cache-registry, so it never writes the registry cache anyway.

Downstream (blocked on the upstream change and the v0 tag update)

  • Add a refresh-cache boolean workflow_dispatch input (default false) and pass refresh-cache: ${{ inputs.refresh-cache || false }} to the shared workflow (inputs.* is null for non-dispatch events; || false keeps schedule/push unchanged):
    • images-connect: production.yml, content.yml
    • images-package-manager: production.yml
    • images-workbench: production.yml, session.yml (both uses: calls)
    • Optional: development*.yml in each repo, for consistency
  • images-specialized: these workflows call bakery build directly, not the shared workflow.
    • wgcw.yml: add a dispatch input that adds --no-cache to the --load build step. Leave the later --push step without --no-cache so it reuses the locally built, tested layers. Note that this workflow passes --cache-registry only to the --load step, so it reads the registry cache but never writes it (bakery only adds cache-to with --push). Find out what refreshes that cache. If nothing does, wgcw builds will keep reading stale layers after this rebuild.
    • positron-sagemaker.yml, rstudio-sagemaker.yml, waml.yml: no registry cache and GitHub-hosted runners start fresh, so a plain dispatch already rebuilds clean. No change needed.

Acceptance criteria

  • A manual dispatch with refresh-cache: true rebuilds every layer, installs the current distro ca-certificates, and refreshes the registry cache entries.
  • Later scheduled and push builds reuse the refreshed cache and don't bring back the old ca-certificates.
  • Scheduled, push, and PR builds behave as before when the flag isn't set.

Activity

  1. added
    cicdContinuous integration/deployment
    securityRelated to security or vulnerability management
    on Oct 21, 2026
  2. changed the title [-]Add a flag to disable build cache in manual workflow dispatches[/-] [+]Add a refresh-cache flag (--no-cache with --cache-registry) for manual workflow dispatches[/+] on Oct 21, 2026
  3. self-assigned this
    on Oct 21, 2026
  4. ianpittwood commented on Oct 21, 2026

    @ianpittwood
    ContributorAuthor

    Downstream PRs, opened as drafts until #850 merges and v0 is updated:

    Remaining for #847: merge #850, update v0, mark these ready, and merge them.

  5. ianpittwood commented on Oct 23, 2026

    @ianpittwood
    ContributorAuthor

    Done. refresh-cache: true runs bakery build --no-cache while keeping --cache-registry, so nothing is read from the build cache and fresh layers are written back. This was verified against a local registry and on the real GHCR cache during the #848 rebuilds.

    Shared workflow:

    Dispatch input in the product repos:

    Related: the recent dispatch input and its 0-parsing fix are attached to #851.

  6. added
    cvp:0Necessary projects we are undertaking that don’t directly deliver value to the customer
    tdp:2The team notices.
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

cicdContinuous integration/deploymentcvp:0Necessary projects we are undertaking that don’t directly deliver value to the customerdockerRelated to container images we producesecurityRelated to security or vulnerability managementtdp:2The team notices.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions