[Bug]: Dockerignore Resolution Order Differs from Docker Proper

[KazWolfe]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

1. Create a new directory somewhere.
2. Create some dummy files:

mkdir -p foo/bar
touch foo/.gitkeep foo/bar/.gitkeep

3. Create a .dockerignore:

/foo/*
!/foo/.gitkeep

/foo/bar/*
!/foo/bar/.gitkeep

4. Create a Dockerfile:

FROM scratch
COPY . .

5. Attempt a build with container build.

Problem description

.dockerignore resolution differs slightly between Docker standard and container build, especially when resolving (un)ignored paths. The provided sample files emit the following error upon building:

container build
[+] Building 0.1s (3/4)                                                                                                                                                                                       
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 58B
 => [internal] load .dockerignore
 => => transferring context: 91
 => ERROR [internal] load build context 
 => => transferring context: 142B
------
 > [internal] load build context:
------
Error: unknown: "failed to solve: changes out of order: "foo/bar/.gitkeep" "foo/.gitkeep""

This causes problems with, e.g., the default Rails .dockerignore configuration as generated by rails new and it leverages this layering to include/exclude certain subpaths while preserving .keeps.

Environment

- OS: macOS 26.5.1 (25F80)
- Xcode: 26.5 (17F42)
- Container: CLI version 1.0.0 (build: release, commit: unspeci); installed via Brew

Code of Conduct

• I agree to follow this project's Code of Conduct

[dyuti518-create]

Hello world I'm Amiyadyuti Ghosh

[Rogerjaytw]

I dug into this and can explain exactly what's happening — the bug is in container-builder-shim, not this repo.

When BuildKit loads the build context, the shim walks the unpacked context tree and filters each entry through the .dockerignore patterns (pkg/fssync/walk.go), dropping excluded entries unconditionally. With this issue's patterns, the directory foo/bar itself is excluded by /foo/* (nothing re-includes the directory), while foo/bar/.gitkeep is re-included by !/foo/bar/.gitkeep. The walk therefore streams foo, foo/.gitkeep, foo/bar/.gitkeep — never emitting the parent directory foo/bar. BuildKit's receiver (fsutil's Validator) requires every path to arrive after its parent directory, and rejects the stream with exactly changes out of order: "foo/bar/.gitkeep" "foo/.gitkeep".

Upstream fsutil's filter handles this case by lazily emitting excluded ancestor directories when a descendant is re-included; the shim's hand-rolled filter doesn't.

I reproduced this in a unit test — on the current shim code, fsutil's Validator rejects the walk output with the exact error above — and opened apple/container-builder-shim#87, which replaces the manual filtering with fsutil.NewFilterFS (the same construction BuildKit's own filesync provider uses), so pattern semantics match Docker exactly. Happy to iterate if the maintainers prefer a different approach.

[chrisgeo]

Hit the same error class on a real-world (non-minimal) Dockerfile — worth noting this isn't limited to the exact re-included-subdirectory pattern in the original report:

Error: unknown: "failed to solve: changes out of order: \"cmd/pkg/somefile.go\" \"Dockerfile\""

• Container: CLI 1.2.2
• builder-shim: ghcr.io/apple/container-builder-shim/builder:0.13.1
• OS: macOS 26.6.2 (Apple silicon)

Our .dockerignore is a deny-all + allowlist pattern (* then !Dockerfile plus a couple of negated globs), i.e. the same "exclude parent, re-include descendant" shape @Rogerjaytw's root-cause analysis describes — just triggered on ordinary source files rather than a .gitkeep. Confirms the bug isn't dockerignore-content-specific, just pattern-shape-specific, and that it's still present as of builder-shim 0.13.1.

Is apple/container-builder-shim#87 (switching to fsutil.NewFilterFS) still the intended fix? Happy to test against a build with that PR's shim image if there's one published.

[Rogerjaytw]

Thanks for offering to test this. I’ve published a temporary, unofficial Linux/arm64 builder image from the current head of apple/container-builder-shim#87:

ghcr.io/rogerjaytw/container-builder-shim/builder:pr87-6df6dfc

Digest:

sha256:cf0296fbaf0ca6cb304de2029c9a8f52347b1af457ba8f5b6aa057485b2f2dc6

It was built from apple/container-builder-shim#87 at 6df6dfc, after syncing with upstream main at e18d218, the commit tagged 0.13.1.

I also ran local A/B tests against both the minimal excluded-parent/re-included-descendant case and a deny-all + allowlist reproduction matching the reported pattern:

• The official builder:0.13.1 image reproduced changes out of order for both foo/bar/.gitkeep and cmd/pkg/somefile.go.
• The PR image completed both builds successfully.
• Inspecting the resulting image layers confirmed that the expected re-included files and their ancestor directories were present at the correct paths.

To test it, set the builder image in ~/.config/container/config.toml:

[build]
image = "ghcr.io/rogerjaytw/container-builder-shim/builder:pr87-6df6dfc"

Then pull the image, delete the existing builder, and retry the original failing build:

container image pull ghcr.io/rogerjaytw/container-builder-shim/builder:pr87-6df6dfc
container builder delete --force
container build ...

container builder delete --force removes the existing builder so the next build will recreate it using the configured image. This may discard the existing builder’s build cache.

This image is only intended for validating apple/container-builder-shim#87 and is not an official Apple build.

Please let me know whether this resolves the original real-world build failure.

[chrisgeo]

Tested the pr87 builder image (ghcr.io/rogerjaytw/container-builder-shim/builder:pr87-6df6dfc, digest sha256:cf0296fbaf0ca6cb304de2029c9a8f52347b1af457ba8f5b6aa057485b2f2dc6, confirmed via docker buildx imagetools inspect) against our real-world failure — thanks for publishing it.

Environment: macOS 26.6.2 (25G83), Apple Silicon, container CLI 1.2.2.

Dockerfile pattern: multi-stage Go build (4 binaries), deny-all .dockerignore with an explicit allowlist of re-included subpaths, plus later COPY --from=<stage> hops feeding two final distroless targets (acr-api, acr-mcp).

Results:

• Official builder:0.13.1: reproduced changes out of order: "<file>" "Dockerfile" on the [internal] load build context step, as expected — this is the [Bug]: Dockerignore Resolution Order Differs from Docker Proper #1800 bug.
• pr87 builder, container build --target acr-api -t ... .: PASS (59.9s cold build). No "changes out of order".
• pr87 builder, full multi-stage build (all targets, no --target): PASS (9.8s, cache-warmed). Exercised three separate COPY --from=<stage> hops across the build graph — no "unsupported offset" from [Bug]: 'failed to compute cache key: failed to copy: unsupported offset' on multi-stage COPY --from copying a whole stage root (builder-shim 0.13.1) #2150 either.
• Ran the resulting acr-api image (acr-api version, acr-api --help) — binary present and functional.

The #1800 failure we hit is resolved by this builder image for our Dockerfile. The full multi-stage build also completed with no unsupported offset error (#2150) across three COPY --from=<stage> hops — noted as an observation only, since the 0.13.1 build fails earlier at context load and so is not an A/B for that issue.

Happy to test further scenarios if useful.

[eimantas]

I've hit the same issue and can confirm that pr87 builder fixed it for me (after restarting the container system).

Environment: macOS 27.0 (26A5421a), Apple Silicon, container CLI version 1.2.2 (build: release, commit: 0190097)