switching from docker.io to docker-ce - #37
Open
vladimirhasko wants to merge 15 commits into
Open
Conversation
vladimirhasko
changed the title
Contributor
Author
|
recheck |
- pod has no ansible_facts: drop fact templates, hardcode debian/bookworm/amd64 for the Docker CE apt repo and GPG URL - pod has no systemd as PID 1: drop systemd/service modules, launch containerd and dockerd as nohup'd background processes and wait for the sockets before verifying with docker info - gather_facts: false (facts unavailable/unnecessary in the pod)
The initial apt update runs before the Docker repo file is written, so the package manager has no knowledge of docker-ce when install fires. Add a second apt update after creating the .list file.
The socket appears early but dockerd hasn't fully initialized yet, causing 'docker info' to fail on first attempt. Replace the single verify task with a 15-attempt retry loop (5s delay, up to 75s total). Also replace 'docker version --format' with 'docker --version' to avoid Jinja2 brace escaping issues.
Replace the silent retry loop with tasks that dump: - docker info output from last attempt - dockerd process list (ps aux | grep dockerd) - last 100 lines of /var/log/dockerd.log - last 100 lines of /var/log/containerd.log Then fail with all that context so we can see what went wrong.
Root cause from the Zuul log: the zuul-debian pod is not privileged (no CAP_NET_ADMIN), so dockerd cannot create the DOCKER NAT chain (iptables 'Permission denied') and exits during network-controller init. The fix is a privileged nodepool label (system-config PR, same pattern as zuul-debian-builder) plus dockerd flags suited to a systemd-less pod: --exec-opt native.cgroupdriver=cgroupfs --iptables=false --ip6tables=false --bridge=none containers will run with host networking (single-node localhost inventory does not need a bridge).
|
recheck |
LukasCuperDT
added a commit
to opentelekomcloud-infra/system-config
that referenced
this pull request
Aug 20, 2026
zuul-debian-docker (prod) and zuul-debian (preprod) declared shell-type: /bin/bash. Zuul passes that to ansible as ansible_shell_type, which is a plugin name, not a path, so every job on those labels died in Gathering Facts with Could not find the shell plugin required (/bin/bash). Ansible ships sh, csh, fish, powershell and cmd - there is no bash plugin; bash is handled by sh, which is what the other three labels already use. Found via stackmon/cloudmon#37, whose job is the first to request zuul-debian-docker from its own playbooks. Our smoke test missed it because the zuul-infra playbooks set ansible_shell_type explicitly.
LukasCuperDT
force-pushed
the
docker_switch
branch
from
August 20, 2026 20:39
b039d04 to
5573681
Compare
* Fix the docker daemon diagnostic block
The 'Fail with diagnostic context' task had no when condition, so the
play failed even when docker started correctly. Conversely the wait_for
above it aborted the play on real failure, so the diagnostics it was
meant to collect never ran in the case they were written for.
Make wait_for non-fatal and put the diagnostics and the failure behind
'docker_sock is failed', so they run exactly when the socket never
appeared.
Also drop the reference to docker_info, which is never registered, and
use pgrep instead of 'ps aux | grep' under ansible.builtin.command,
which does not interpret pipes - those characters were being passed to
ps as literal arguments.
* Install python packaging and venv tooling before cloudmon roles
install_cloudmon failed on the zuul-debian-docker node with:
ModuleNotFoundError: No module named 'packaging'
ansible.builtin.pip imports packaging on the target, and cloudmon_venv_path
defaults to ~/.venv so the module also has to build a virtualenv. The node
image provides neither, so install them before the roles run.
* Surface cloudmon provision output instead of a bare rc
The Deploy task ran graphite, statsd and postgres provisioning in one shell
block with no 'set -e', so rc came from the last command only and an earlier
failure would pass unnoticed. On failure Ansible reported just 'non-zero return
code' with no stdout/stderr, and job-output.json is not uploaded, so there was
nothing to debug from.
Run one component per loop item, print rc/stdout/stderr for each, then fail
listing the components that actually failed.
* Fix inventory template: group hosts must be a mapping
Every child group declared its hosts as a list:
statsd:
hosts:
- localhost
Ansible's yaml inventory plugin requires a mapping there and rejected the
whole file:
Invalid "hosts" entry for "statsd" group, requires a dictionary
With no inventory parsed, every provisioning play matched no hosts. graphite
still exited 0 while doing nothing ('skipping: no hosts matched'), so the
failure surfaced only as statsd/postgres rc=1.
Reproduced with ansible-inventory: the old template raises the exact error,
the new one resolves all 8 groups to localhost.
* Use local connection for localhost in cloudmon inventory
Provisioning failed with:
ssh: connect to host localhost port 22: Connection refused
fatal: [localhost]: UNREACHABLE!
Listing localhost explicitly in the inventory overrides Ansible's implicit
local connection, so it fell back to ssh. The zuul-debian pod runs no sshd,
so set ansible_connection: local.
* Allow skipping host firewall management
Graphite provisioning failed in the pod-based job with:
Could not find the requested service firewalld: host
The firewalld role is pulled in unconditionally by seven install playbooks and
manages host services. A container has no service manager and no host
firewall, so those tasks cannot succeed. 'Disable iptables' already had
ignore_errors; 'Enable firewalld' did not, so it aborted the play.
Gate the role body behind cloudmon_manage_firewall, defaulting to true so
VM-based deployments are unchanged, and set it false in the k8s job
inventory.
* Use the block keyword instead of ansible.builtin.block
Provisioning aborted with:
unexpected parameter type in action: <class ..._AnsibleTaggedList>
Origin: roles/graphite/tasks/main.yaml:38:3
'block' is a task keyword, not a module -- there is no ansible.builtin.block
action, so ansible-core treats it as a module receiving a list.
Fixes all 22 occurrences across 11 roles, introduced in 685afcf.
* Use netcat-openbsd on Debian
Package installation failed on the Debian bookworm node:
E: Package 'netcat' has no installation candidate
Package netcat is a virtual package provided by:
netcat-openbsd 1.219-1
netcat-traditional 1.10-47
'netcat' is virtual on bookworm and cannot be installed directly. Use
netcat-openbsd, which provides nc, in the graphite, statsd and carbonapi
Debian vars.
* Gate per-role firewalld tasks too
Graphite provisioning still failed after gating the firewalld role:
TASK [graphite : Allow graphite port]
Failed to import the required Python library (firewall)
Origin: roles/graphite/tasks/firewall.yml:4:3
Six roles include their own firewall.yml to open ports via ansible.posix
.firewalld, which needs the firewall python library and a running firewalld.
Neither exists in a container, so gate these includes behind the same
cloudmon_manage_firewall variable.
Graphite provisioning failed with:
Failed to find required executable "crontab" in paths: ...
The rewritten Debian vars dropped 'cron', which main still lists. The graphite
role is the only one using ansible.builtin.cron ('Run periodic cleanup'), and
that module needs the crontab binary.
apparmor-profiles was dropped in the same rewrite but is never referenced by
any task, so it stays out.
|
recheck |
Provisioning failed once it reached the application layer:
System has not been booted with systemd as init system (PID 1).
Can't operate. Failed to connect to bus: Host is down
cloudmon deploys through systemd -- 15 unit templates, 68 systemd tasks and 14
daemon_reload calls -- so the node needs a real init. Switch the nodeset to the
new zuul-debian-systemd label, which runs systemd as PID 1.
With systemd present, let it own containerd and dockerd instead of starting
them with nohup; hand-started daemons would collide with the units the
cloudmon roles enable. The daemon settings now come from /etc/docker/daemon.json
seeded by the pod spec. Diagnostics read the journal instead of the log files
the manual start used to write.
Verified on a probe pod on the cluster: PID 1 is systemd, is-system-running
reports running, daemon-reload succeeds, and a unit written at runtime can be
enabled and started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.