Reliable and automated base image build process by nccatoni · Pull Request #7270 · DataDog/system-tests

nccatoni · 2026-07-06T16:00:30Z

Content-Addressed Base Images via GitLab CI

Summary

Automate weblog base image publishing with content-addressed tags, replacing the manual label-driven workflow (build-*-base-images PR labels + manual version bumps + human push coordination).

What Changed

New build_base_images GitLab CI job runs on every push: derives dependencies from COPY instructions in each *.base.Dockerfile, computes a 12-char content hash, and pushes datadog/system-tests:<name>.base-<hash> if the tag doesn't exist. Idempotent — never overwrites.
Lock file (utils/build/docker/base-images.lock.json) maps stable aliases (system_tests_base_nodejs_express4) → immutable tags. Consumer Dockerfiles use FROM <alias>, resolved via BuildKit --build-context at build time.
GitHub Actions no longer builds base images — polls Docker Hub (wait_for_base_image.py) for the locked tag instead.
Removed: per-language build scripts, build-*-base-images labels, _build_weblog_base_images workflow input, manual -vN tag management.

New Workflow

Edit a base Dockerfile or its dependencies → push
GitLab CI auto-publishes the new content-addressed tag
Run python utils/scripts/build_base_images.py --update-lock + python utils/scripts/update_mirror_images.py
Commit the updated lock/mirror files

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-07-06T16:01:05Z

CODEOWNERS have been resolved as:

tests/test_the_test/test_build_base_images.py                           @DataDog/system-tests-reviewers
utils/base_images/base_image.py                                         @DataDog/system-tests-core
utils/base_images/build_base_images.py                                  @DataDog/system-tests-core
utils/base_images/wait_for_base_image.py                                @DataDog/system-tests-core
utils/build/docker/base-images.lock.json                                @DataDog/system-tests-reviewers
utils/scripts/resolve-base-images-contexts.py                           @DataDog/system-tests-core
utils/scripts/update-base-images-lock.py                                @DataDog/system-tests-core
utils/scripts/wait-for-base-images.py                                   @DataDog/system-tests-core
.dockerignore                                                           @DataDog/system-tests-core
.github/workflows/ci.yml                                                @DataDog/system-tests-core
.github/workflows/compute-workflow-parameters.yml                       @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
.github/workflows/system-tests.yml                                      @DataDog/system-tests-core
.gitignore                                                              @DataDog/system-tests-core
.gitlab-ci.yml                                                          @DataDog/system-tests-core
docs/CI/github-actions.md                                               @DataDog/system-tests-core
docs/edit/update-docker-images.md                                       @DataDog/system-tests-core
docs/understand/weblogs/weblog-metadata.md                              @DataDog/system-tests-core
mirror_images.lock.yaml                                                 @DataDog/system-tests-core
mirror_images.yaml                                                      @DataDog/system-tests-core
pyproject.toml                                                          @DataDog/system-tests-core
tests/test_the_test/test_ci_orchestrator.py                             @DataDog/system-tests-reviewers
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/_context/weblog_metadata.py                                       @DataDog/system-tests-core
utils/build/build.sh                                                    @DataDog/system-tests-core
utils/build/docker/nodejs/docker-bake.hcl                               @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express4-typescript.Dockerfile                @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express4-typescript.base.Dockerfile           @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express4.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express4.base.Dockerfile                      @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express5.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express5.base.Dockerfile                      @DataDog/system-tests-reviewers
utils/build/docker/nodejs/fastify.Dockerfile                            @DataDog/system-tests-reviewers
utils/build/docker/nodejs/fastify.base.Dockerfile                       @DataDog/system-tests-reviewers
utils/build/docker/nodejs/nextjs.Dockerfile                             @DataDog/system-tests-reviewers
utils/build/docker/nodejs/nextjs.base.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/nodejs/uds-express4.Dockerfile                       @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.0-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.0.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.1-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.1.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.2-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.2.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.3-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.3.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.4-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-7.4.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-8.0-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-8.0.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-8.1-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-8.1.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-8.2-zts.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod-8.2.Dockerfile                        @DataDog/system-tests-reviewers
utils/build/docker/php/apache-mod.base.Dockerfile                       @DataDog/system-tests-reviewers
utils/build/docker/php/docker-bake.hcl                                  @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-7.0.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-7.1.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-7.2.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-7.3.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-7.4.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-8.0.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-8.1.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-8.2.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm-8.5.Dockerfile                           @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm.base.Dockerfile                          @DataDog/system-tests-reviewers
utils/build/docker/php/php-fpm/build.sh                                 @DataDog/system-tests-reviewers
utils/build/docker/php/weblog_metadata.yml                              @DataDog/system-tests-reviewers
utils/build/docker/python/django-poc.Dockerfile                         @DataDog/system-tests-reviewers
utils/build/docker/python/django-poc.base.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/python/django-py3.13.Dockerfile                      @DataDog/system-tests-reviewers
utils/build/docker/python/django-py3.13.base.Dockerfile                 @DataDog/system-tests-reviewers
utils/build/docker/python/docker-bake.hcl                               @DataDog/system-tests-reviewers
utils/build/docker/python/fastapi.Dockerfile                            @DataDog/system-tests-reviewers
utils/build/docker/python/fastapi.base.Dockerfile                       @DataDog/system-tests-reviewers
utils/build/docker/python/flask-poc.Dockerfile                          @DataDog/system-tests-reviewers
utils/build/docker/python/flask-poc.base.Dockerfile                     @DataDog/system-tests-reviewers
utils/build/docker/python/python3.12.Dockerfile                         @DataDog/system-tests-reviewers
utils/build/docker/python/python3.12.base.Dockerfile                    @DataDog/system-tests-reviewers
utils/build/docker/python/tornado.Dockerfile                            @DataDog/system-tests-reviewers
utils/build/docker/python/tornado.base.Dockerfile                       @DataDog/system-tests-reviewers
utils/build/docker/python/uds-flask.Dockerfile                          @DataDog/system-tests-reviewers
utils/build/docker/python/uwsgi-poc.Dockerfile                          @DataDog/system-tests-reviewers
utils/build/docker/python/uwsgi-poc.base.Dockerfile                     @DataDog/system-tests-reviewers
utils/build/docker/python_otel/flask-poc-otel.Dockerfile                @DataDog/system-tests-reviewers
utils/ci/gitlab/main.yml                                                @DataDog/system-tests-core
utils/scripts/ci_orchestrators/workflow_data.py                         @DataDog/system-tests-core
utils/scripts/compute-workflow-parameters.py                            @DataDog/system-tests-core
utils/scripts/get_pr_merged_labels.sh                                   @DataDog/system-tests-core
utils/scripts/libraries_and_scenarios_rules.yml                         @DataDog/system-tests-core
utils/build/build_nodejs_base_images.sh                                 @DataDog/system-tests-core
utils/build/build_php_base_images.sh                                    @DataDog/system-tests-core
utils/build/build_python_base_images.sh                                 @DataDog/system-tests-core
utils/scripts/build-base-image.py                                       @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-07-06T16:39:14Z

⚠️ Warnings

⚠️ Your PR has warnings. Please review the issues below.

🚦 1 Pipeline job failed

Testing the test | all-jobs-are-green

View more details · View in GitHub Actions

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ee4b7b9 | Docs | View more details | Give us feedback!}

# Conflicts: # mirror_images.lock.yaml # mirror_images.yaml # utils/_context/containers.py # utils/build/docker/python/django-poc.Dockerfile # utils/build/docker/python/django-poc.base.Dockerfile # utils/build/docker/python/django-py3.13.Dockerfile # utils/build/docker/python/django-py3.13.base.Dockerfile # utils/build/docker/python/docker-bake.hcl # utils/build/docker/python/fastapi.Dockerfile # utils/build/docker/python/fastapi.base.Dockerfile # utils/build/docker/python/flask-poc.Dockerfile # utils/build/docker/python/flask-poc.base.Dockerfile # utils/build/docker/python/python3.12.Dockerfile # utils/build/docker/python/python3.12.base.Dockerfile # utils/build/docker/python/tornado.Dockerfile # utils/build/docker/python/tornado.base.Dockerfile # utils/build/docker/python/uds-flask.Dockerfile # utils/build/docker/python/uwsgi-poc.Dockerfile # utils/build/docker/python/uwsgi-poc.base.Dockerfile

cbeauchesne

One question, and one non-blocking suggestion.

So excited by this PR !

nccatoni force-pushed the nccatoni/base-image-build branch 2 times, most recently from e2c8ca6 to 611f535 Compare July 6, 2026 16:25

nccatoni force-pushed the nccatoni/base-image-build branch 2 times, most recently from dc70509 to b8f1ceb Compare July 7, 2026 11:52

nccatoni force-pushed the nccatoni/base-image-build branch from d3cc9d8 to 1902e53 Compare July 30, 2026 10:14

nccatoni added 6 commits July 30, 2026 12:31

feat(base-images): build content-addressed images in GitLab

90e8984

ci: wait for GitLab-built weblog base images

d935f5a

docs(base-images): document automated image publishing

5ce2f00

fix(php): install mbstring in php-fpm base images

3e66c2c

chore(base-images): use content-addressed tags

6a9e4c9

feat(base-images): resolve consumers through locked contexts

578a92d

nccatoni force-pushed the nccatoni/base-image-build branch from 1902e53 to 578a92d Compare July 30, 2026 12:06

nccatoni added 2 commits July 30, 2026 15:10

More reliable hashes

1a1bb3a

Lock file should not be drift checked

5d90e00

nccatoni force-pushed the nccatoni/base-image-build branch from f90836f to 5d90e00 Compare July 30, 2026 13:44

Update gitlab mirroring lock file

5ac604f

nccatoni marked this pull request as ready for review July 31, 2026 13:34

nccatoni requested review from a team as code owners July 31, 2026 13:34

nccatoni requested review from Kyle-Verhoog, christophe-papazian and dubloom and removed request for a team July 31, 2026 13:34

nccatoni added the changelog label Jul 31, 2026

gh-worker-ownership-write-b05516 Bot removed the request for review from a team August 26, 2026 16:32

nccatoni added 8 commits August 26, 2026 18:39

Lock script fix and lock update

556f60c

Update docs and comments to reference renamed base-image scripts

5f24f1f

docs: document base image copy sources

677aa8b

fix: fail fast while waiting for base images

79e9f4f

chore: remove obsolete base image workflow flag

cdb8cde

chore: pluralize base image tooling

78bf968

format

b6a3e19

nccatoni force-pushed the nccatoni/base-image-build branch from de366be to b6a3e19 Compare September 9, 2026 09:35

Update locks

2546794

nccatoni force-pushed the nccatoni/base-image-build branch 2 times, most recently from 561db38 to 16226c7 Compare September 9, 2026 14:07

Solving import issues

7668b2f

nccatoni force-pushed the nccatoni/base-image-build branch 2 times, most recently from 068950d to f28b8cb Compare September 9, 2026 15:04

Fix weblogs without docker files

bf2ed4b

nccatoni force-pushed the nccatoni/base-image-build branch from f28b8cb to bf2ed4b Compare September 9, 2026 15:20

nccatoni requested a review from cbeauchesne September 9, 2026 16:08

Update image mirror lock file

15ca33f

cbeauchesne approved these changes Sep 10, 2026

View reviewed changes

Comment thread .github/workflows/system-tests.yml

Comment thread pyproject.toml Outdated

nccatoni added 9 commits September 10, 2026 14:12

Merge branch 'main' into nccatoni/base-image-build

51d5a92

Merge branch 'main' into nccatoni/base-image-build

844a315

Replace print with logger

9244bd2

Lock update

b8855a6

Merge branch 'main' into nccatoni/base-image-build

a12b7ed

Merge branch 'main' into nccatoni/base-image-build

567321d

Base image lock update

79765aa

Image mirror update

a5dfcef

Fix mirror update

ee4b7b9

Navigation Menu

Uh oh!

Reliable and automated base image build process - #7270

Reliable and automated base image build process#7270
nccatoni wants to merge 46 commits into
mainfrom
nccatoni/base-image-build

nccatoni commented Jul 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 6, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

cbeauchesne left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nccatoni commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content-Addressed Base Images via GitLab CI

Summary

What Changed

New Workflow

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 6, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nccatoni commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 6, 2026 •

edited by datadog-official Bot

Loading