Ansible for a Workstation

A deliberately boring foundation under a fast-moving ecosystem.

Ansible control repository for provisioning Fedora 44, AlmaLinux 10, and Rocky Linux 10 machines — system configuration, virtualisation, and container runtimes — from a single inventory with verified support for Ansible core 2.18–2.20 and Python 3.14.

Kept as a working notebook: the roles, playbooks, and plugins are the current entries of a decade's experiments, and the git history keeps the earlier entries.

Introduction

Every distribution family is a philosophy wearing a package manager.

The Enterprise Pragmatists — RHEL, Rocky, Alma — sell predictability. Software should not change for ten years; a bug that breaks a feature is bad, but an update that changes how a feature behaves is a catastrophe. The Upstream Innovators — Fedora, Arch, Tumbleweed — run the other way: standing still is falling behind, and Fedora exists to test what will land in RHEL five years from now. Choosing between them was never really a question of package managers. It is a question of alignment: does the worldview of the people building the operating system match the work you are trying to do on top of it.

Most of my career sat on the upstream side of that line, where the technical conversation is actively happening. This repository is the record of stepping across to the pragmatic ground. Not a rejection of the upstream mindset — a calculated structural alignment. Outsource the churn of the base operating system, and the creative bandwidth goes to what runs on top of it instead of to fighting the shifting sands underneath it.

Then the LLM ecosystem moved the spotlight onto exactly this choice. When the application layer stops executing predictable, hard-coded logic and starts running non-deterministic prompts, autonomous agent loops, and dynamic tool calls, the host operating system can no longer afford to be a moving target. A multi-agent system orchestrates file parsers, Python runtimes, and external APIs; if the base OS changes behavior under it during a routine update, debugging a rogue tool call becomes archaeology. Specialized compute is fragile enough on its own — CUDA toolkits and GPU drivers against kernel namespaces — without the floor moving too.

The machinery that finally made this cheap is recent. The constraint is not. bootc, image-based updates, and transactional rollback only reached usable shape a few years ago, while the constraint itself — a fast-moving workload needs a floor that does not move — was learned the slow way, across a decade of CI/CD pipelines and realtime audio workstations. Adopting the image tooling was not a change of thesis. It was the ecosystem finally shipping the machinery the thesis always needed.

So this repository runs at two speeds.

The slow layer is deliberately boring: idempotent roles, explicit variable tiers, distributions asserted before anything touches the disk, firewall rules co-located with the services that need them. Ansible's whole job here is to keep this layer honest — re-running a playbook against a converged host should report zero changes.

The fast layer is everything that legitimately churns: language models, agent harnesses, RAG pipelines, tokenizers, the tooling that reinvents itself weekly. That layer lives in containers and disposable images, walled off from the host's ABI, free to be reckless precisely because the ground underneath it is not.

The name is borrowed from music. Syncopation puts the accent where the meter does not expect it — between the beats, against the pulse — and it only works because the pulse holds. Nothing in this repository keeps the same time. The AlmaLinux 10 and Rocky Linux 10 base moves on a ten-year clock, Fedora 44 on a thirteen-month one, Ansible (core 2.18–2.20 on Python 3.14) whenever a playbook runs, the agent tooling on whatever upstream shipped this week. Syncopated is the practice of letting those disparate systems land on their own offbeats without losing the downbeat underneath them.

Make yourself comfortable. The plumbing below is just the argument, load-bearing.

What Exists Today

This repository is a notebook. I use it to test current tools and methods against real hosts and to keep a record of what I found. Nothing in it is finished; each role, playbook, and plugin is the current entry for an experiment, and the git history keeps the earlier entries. Practically, it renders the machines in its inventory from a single declaration — but the machines are the current instantiation, not the point. The repository is an archive of blueprints — methods, patterns, known-good configurations — that can be picked up and pointed at any host list, not only mine.

The state of the entries, facet by facet:

• Image builds live in a dedicated standalone repository. The osbuild machinery — composing Fedora, AlmaLinux, and Rocky Linux ISOs, appliances, and container disks from declarative blueprints — has been extracted to the standalone osbuild repository alongside the package taxonomy and package_tracker.py management tooling. Repository GPG keys are fetched locally and verified before any compose; the target distribution is asserted before anything touches the host. The base image remains the reproducible artifact floor beneath this fleet, decoupled so image composition evolves without entangling host configuration runs.
• RPM package development lives in a dedicated standalone repository. The rpm_dev role, mock build chroots, and packaging environments have been extracted to the standalone rpmbuild repository. Package specs and repo builds iterate in isolated mock containers without polluting workstation state.
• LLMOps and remediation live in a dedicated standalone repository. Dedicated AI application platforms (dify, langfuse), agent runtimes (hermes, ollama), their dedicated service playbooks, and the autonomous LLM diagnose-and-remediate pipeline with mechanical safety guards have been extracted to the standalone llmops repository. Decoupled from host provisioning, the LLM runtime and its self-healing pipeline evolve alongside model updates without risking the workstation baseline.
• Full workstation provisioning runs on Fedora 44, AlmaLinux 10, and Rocky Linux 10.
  • playbooks/workstation.yml includes four plays — base, system, user, coding agents — against the workstations group in inventory for multi-host push provisioning.
  • local.yml provides an autonomous local pull and convergence playbook specifically designed for ansible-pull and local bootstrap. It implements the 15 Ansible Design Patterns with pre-flight hardware auto-detection (KVM, graphical display, containers, CUDA/Vulkan GPU acceleration), Tag-as-Select role filtering, check-mode safety, and post-flight reboot verification. Execution is decoupled into two discrete plays: Play 1 (system, become: true) handles privileged hardware and OS foundation, while Play 2 (user, become: false) provisions unprivileged user environment and uv Python toolchains without prompting for sudo.
  • Staged bootstrap lifecycle orchestrates automated convergence from fresh minimal installations:
    • Stage 1 (scripts/bootstrap.sh): Standalone shell script installing minimal prerequisites (git, ansible-core, python3-dnf, python3-rich), running a full dnf upgrade -y --refresh, staging syncopated-bootstrap-root.service, and triggering an initial reboot.
    • Stage 2 (scripts/syncopated-bootstrap-root.sh): Automated post-reboot systemd runner configured with TimeoutSec=0 executing ansible-pull --tags system local.yml, writing completion sentinel /var/log/syncopated-stage2.done, self-disabling, and rebooting into the graphical environment.
    • Stage 3 (syncopated-firstboot): Deployed system-wide by roles/base with autostart desktop entries and profile hooks; triggers upon first user login to clone dotfiles via native yadm, run unprivileged user bootstrap, and mark ~/.local/state/syncopated/firstboot.done.
  • Graphical target & native Wayland GDM default on workstations: roles/desktop enables desktop provisioning by default (enable_desktop: true), configures GDM with native Wayland (WaylandEnable=true in /etc/gdm/custom.conf), promotes the systemd default target to graphical.target, and installs native RPMs (such as input-remapper on Fedora).
  • Native RPM packaging in the Syncopated COPR (b08x/syncopated): delivers clean, native RPMs for yadm, gitflow, gum, and uv across Fedora and Enterprise Linux 10, eliminating reliance on unverified GitHub downloads or user-space curl pipelines.
  • Integrated with Yadm bootstrap via 01-ansible-pull.sh, which acts as an automated orchestration bridge: on fresh machine onboarding, Yadm triggers 01-ansible-pull.sh to converge Layer 1 host packages and services via local.yml before executing user dotfiles (Layer 2).
  • A strict distribution taxonomy is codified in docs/layering-architecture.md, partitioning duties across OSBuild (Layer 0), Ansible (Layer 1), and Yadm (Layer 2). Detailed bootstrap mechanisms are documented in docs/bootstrap.md.
  • Local virtualization testing via Vagrantfile: provides a dedicated libvirt Fedora workstation VM definition (vagrant up fedora) supporting customizable box releases (FEDORA_BOX) for rapid, isolated testing.
  • All 10 roles frontload pre-flight assertions (tasks/assert.yml) to fail fast (< 1s) on unmet preconditions (OS family, DNF/DNF5 package manager, effective root elevation via id -u, and identity variables) before mutations occur.
  • System-level utilities (bun and GPU-gated transcribe.cpp supporting CUDA and Vulkan backends) live natively in roles/tools. Transcribe compilation is skipped automatically on hosts without detected GPU devices (tools_transcribe_require_gpu: true).
• Observability runs in both directions. The llm_analyzer callback records what every play and task actually does to an append-only trace store, and scripts/llm_trainset.py turns those traces into a scored DSPy trainset; rich-ci-formatter renders the same runs for the human at the terminal — DNF5 transaction tables, systemd transitions, phase banners. The formatter works; its failure path is the owed polish: a failed task currently obscures more than it shows.
• Testing exists where the risk lives. The base role carries a Molecule scenario that builds Fedora and AlmaLinux containers, converges twice, and fails on any second-run change. Everything lints.
• bootc is direction, not state. The image-transaction path is designed for, walked partway; the dnf baseline carries the weight today.

What is owed is polish, not function: check-mode idempotency, fail-fast assertions (< 1s), and end-to-end autonomous convergence via local.yml and Yadm 01-ansible-pull.sh are fully verified across Fedora 44, AlmaLinux 10, and Rocky Linux 10. The remaining polish is in the rich-ci-formatter failure path and a recorded end-to-end demonstration. The evidence so far is convergence on real hosts and the trace store, not a screencast.

Intent

One decision, applied consistently, is the whole architecture: the base does not move, so everything above it is allowed to. The intents are recorded explicitly, layer by layer, because the parts only make sense as a whole.

Layers

A strict three-tier taxonomy governs distribution across tooling (see docs/layering-architecture.md):

• The image — the floor (Layer 0). Kernel, drivers, container runtimes, and the base package set. Built, never mutated in place; a change is a new artifact with rollback, not a live edit to a running system. Declarative blueprint generation and image composing are maintained in the companion osbuild repository.

• Ansible — the system above the floor (Layer 1). Host-level packages (base_packages), system services, hardware acceleration, system utilities and coding agent CLIs (roles/tools), and system-wide Flatpaks (roles/desktop). Idempotent by contract — re-running a playbook against a converged host should report zero changes.

• yadm — the user (Layer 2). Dotfiles, shell environment (~/.zshrc), user version managers (rbenv, rustup), default Ruby gems, user Cargo crates, and user-space Flatpaks (80-install-user-flatpaks.sh). The one layer a re-image does not erase, and therefore the one layer where drift is permitted — under version control, where a change is recoverable and attributable.

• Containers — the fast layer. Everything that legitimately churns — language models, agent harnesses, RAG pipelines, tokenizers — runs inside its own dependency closure and never touches the host ABI. Dedicated containerized service stacks (Dify, Langfuse, Hermes, Ollama) and remediation tooling are managed in the companion llmops repository, free to be reckless precisely because the ground underneath them is not.

Software arrival, in three tiers

One source of truth for how software arrives:

• OS packages — dnf, pacman, apt. Their versions are the distribution's problem; that is the service being bought.
• Language ecosystems — npm, uv, cargo, gem. An explicit version decision: pin to the system toolchain, or bring a newer one into user space. Both are fine. Half of each is not.
• OSS projects — git clones and release tarballs, pinned by revision or checksum, never by hope.

The practice is deciding what belongs on which layer — and then holding the line when the next shiny thing arrives and demands to be installed directly onto the host.

Loops

Three pieces of tooling close a loop around the fleet. Every point where the loop acts is gated by a human:

• Observe — llm_analyzer and rich-ci-formatter. Two result callbacks, two audiences: one explains the run to a model and records it to the trace store; one explains the run to the human at the terminal. Agents drift without grounding; humans do too. The trace is the grounding for the machine; the rendered log, for you.
• Remediate — llmops/playbooks/remediate.yml. Operating from the companion llmops repository, the remediation pipeline reads a failing host's state across the fleet, asks a model for a diagnosis and a draft playbook, checks the draft mechanically — guards reject a draft that removes a volume, or replaces a working container without proving its replacement first — and executes nothing without an explicit yes. Confirmed fixes are indexed as reference for the next similar failure on any host.
• Learn — scripts/llm_trainset.py. Turns the accumulated traces into a DSPy trainset and scores it without running a playbook.

Agents provision the machines that watch agents provision machines. The loop is permitted to exist because the layer beneath it does not move.

If you came for the workstation plumbing, read straight down. The intent above is only the reason the plumbing looks like this.

Lineage

This repository is not the first body this exercise has worn. The archive began on the pragmatic ground, wandered upstream, and returned — the workload changing underneath it as it went:

• Nashville (2014–2015) — a startup's CI/CD pipeline on CentOS 7: Jenkins, GitLab, MariaDB with GTID, Nginx, Redis, HAProxy, CIS hardening, three environments, twenty-one roles. The thesis was already whole in the notes from back then: no one should ever have to log into these hosts — every change flows through Ansible.
• Columbus (2016) — the break. Several years of high-stress DevOps work, then a deliberate exit and a turn to the Linux audio ecosystem — JACK, realtime privileges, low-latency tuning. The experimental role collection that follows grew out of this period.
• ArchLabs — the upstream excursion: a complete Arch playbook with a dedicated realtime-audio role, hosts managing themselves with ansible-pull. soundbot enters the inventory here and has never left.
• pop!_OS — the middle ground, tried and found to be someone else's philosophy.
• Fedora (Ansible_RAG) — the archive itself: every Ansible tree from those years, parsed into 1,400+ task files across 68 module buckets — the raw material for deciding which patterns become a retrievable corpus and which get discarded.

The workload changed — enterprise CI/CD first, realtime audio after — but the tension underneath did not, and the two workloads enforced opposite poles of it. The Nashville pipeline is the immutability pole: cattle, hardened, no human hands on the machines. The audio workstation is the other pole: an instrument, hand-tuned mid-session, where re-rendering the machine from a declaration is the last thing anyone wants. Attempts to declare all of it fight the user; attempts to declare none of it lose the machine. Those are also, not coincidentally, the two populations any studio-scale pipeline has to keep alive at once.

The layer split is the settled compromise: make immutable what can be immutable, manage what belongs to the system, and let the rest drift — under version control, where a change is recoverable and attributable. Drift with a changelog beats drift without one, and it beats enforced stasis too.