Fix cloudmon-deploy-zuul-debian job - #38
Merged
Merged
Conversation
The 'Fail with diagnostic context' task had no when condition, so the play failed even when docker started correctly. Conversely the wait_for above it aborted the play on real failure, so the diagnostics it was meant to collect never ran in the case they were written for. Make wait_for non-fatal and put the diagnostics and the failure behind 'docker_sock is failed', so they run exactly when the socket never appeared. Also drop the reference to docker_info, which is never registered, and use pgrep instead of 'ps aux | grep' under ansible.builtin.command, which does not interpret pipes - those characters were being passed to ps as literal arguments.
install_cloudmon failed on the zuul-debian-docker node with: ModuleNotFoundError: No module named 'packaging' ansible.builtin.pip imports packaging on the target, and cloudmon_venv_path defaults to ~/.venv so the module also has to build a virtualenv. The node image provides neither, so install them before the roles run.
The Deploy task ran graphite, statsd and postgres provisioning in one shell block with no 'set -e', so rc came from the last command only and an earlier failure would pass unnoticed. On failure Ansible reported just 'non-zero return code' with no stdout/stderr, and job-output.json is not uploaded, so there was nothing to debug from. Run one component per loop item, print rc/stdout/stderr for each, then fail listing the components that actually failed.
Every child group declared its hosts as a list:
statsd:
hosts:
- localhost
Ansible's yaml inventory plugin requires a mapping there and rejected the
whole file:
Invalid "hosts" entry for "statsd" group, requires a dictionary
With no inventory parsed, every provisioning play matched no hosts. graphite
still exited 0 while doing nothing ('skipping: no hosts matched'), so the
failure surfaced only as statsd/postgres rc=1.
Reproduced with ansible-inventory: the old template raises the exact error,
the new one resolves all 8 groups to localhost.
Provisioning failed with:
ssh: connect to host localhost port 22: Connection refused
fatal: [localhost]: UNREACHABLE!
Listing localhost explicitly in the inventory overrides Ansible's implicit
local connection, so it fell back to ssh. The zuul-debian pod runs no sshd,
so set ansible_connection: local.
Graphite provisioning failed in the pod-based job with:
Could not find the requested service firewalld: host
The firewalld role is pulled in unconditionally by seven install playbooks and
manages host services. A container has no service manager and no host
firewall, so those tasks cannot succeed. 'Disable iptables' already had
ignore_errors; 'Enable firewalld' did not, so it aborted the play.
Gate the role body behind cloudmon_manage_firewall, defaulting to true so
VM-based deployments are unchanged, and set it false in the k8s job
inventory.
Provisioning aborted with:
unexpected parameter type in action: <class ..._AnsibleTaggedList>
Origin: roles/graphite/tasks/main.yaml:38:3
'block' is a task keyword, not a module -- there is no ansible.builtin.block
action, so ansible-core treats it as a module receiving a list.
Fixes all 22 occurrences across 11 roles, introduced in 685afcf.
Package installation failed on the Debian bookworm node:
E: Package 'netcat' has no installation candidate
Package netcat is a virtual package provided by:
netcat-openbsd 1.219-1
netcat-traditional 1.10-47
'netcat' is virtual on bookworm and cannot be installed directly. Use
netcat-openbsd, which provides nc, in the graphite, statsd and carbonapi
Debian vars.
Graphite provisioning still failed after gating the firewalld role:
TASK [graphite : Allow graphite port]
Failed to import the required Python library (firewall)
Origin: roles/graphite/tasks/firewall.yml:4:3
Six roles include their own firewall.yml to open ports via ansible.posix
.firewalld, which needs the firewall python library and a running firewalld.
Neither exists in a container, so gate these includes behind the same
cloudmon_manage_firewall variable.
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for
cloudmon-deploy-zuul-debian, which failed on every run of #37. Each bug was masked by the previous one, so they had to be peeled back one at a time.Targets
docker_switchso it merges into #37.062138cwait_forwas fatal, so it aborted the play before the diagnostic block could run — and thefailtask had no condition, so it fired even on success.f0bd65fansible.builtin.pipimportspackagingon the target andcloudmon_venv_pathmakes it build a venv. The node image ships neither.b600793set -e, sorccame from the last command only and output was never surfaced. Failures appeared as a barenon-zero return code;job-output.jsonis not uploaded, so there was nothing to debug from.ff9e113hostsas a list. Ansible requires a mapping and rejected the whole file, so every play matched no hosts — graphite still exited0while doing nothing.6c96208localhostexplicitly loses the implicit local connection, so Ansible fell back to ssh. The pod runs no sshd.4c19b40firewalldrole manages host services; a container has no service manager.Disable iptableshadignore_errors,Enable firewallddid not. Gated behindcloudmon_manage_firewall(defaulttrue, so VM deployments are unchanged).83b3bd6ansible.builtin.blockis not a module —blockis a task keyword. 22 occurrences across 11 roles, from685afcf.b039d04netcatis a virtual package on Debian bookworm and cannot be installed directly. Usesnetcat-openbsd.Items 3, 7 and 8 are bugs in the branch itself rather than environment issues — the job never ran far enough to reach them.
Verification done rather than assumed: the inventory change was checked with
ansible-inventoryin both directions (old template reproduces the exactrequires a dictionaryerror, new one resolves all 8 groups), andcloudmon_manage_firewall=Falsewas confirmed to resolve on the host.The job still fails after these — see the PR discussion for the current point of failure.
Note on #6: this touches
cloudmon/ansible/project/, which is pip-installed and shipped to anyone deploying cloudmon on real hosts, so I gated it rather than removing it. If VM deployments are truly gone, thefirewalldrole, thecloudmon-deploy/cloudmon-deploy-ubuntu-jammyjobs andplaybooks/should be removed together in a separate change.