diff --git a/proposals/architecture-overview.md b/proposals/architecture-overview.md new file mode 100644 index 0000000..03fefed --- /dev/null +++ b/proposals/architecture-overview.md @@ -0,0 +1,148 @@ +# OALS Architecture Proposal — Overview + +**Status:** Proposal, not implemented, not yet reviewed +**Author:** Jonathan Reichardt +**Version:** Draft 0.1 +**Last updated:** 2026-05-31 +**Audience:** OALS contributors (current and future) +**Reading order:** this overview, then the three documents in the order given in [§2](#2-reading-order-and-why). +**Prerequisites:** general familiarity with the OALS codebase and with OAN (`OpenAudioNetwork/`) at the level of `common/packet_structs.h` and `netutils/LowLatSocket.h`. + +## Abstract + +This document is the entry point for a three-part architecture proposal that I +am putting forward for OALS. The three parts together describe a proposed +rework of how a single engine processes audio, how possibly multiple devices coordinate +into a cluster, and the wire-level protocol that makes cluster-wide atomic +reconfiguration possible. None of the three parts has been implemented. The purpose of this overview is +to frame the three together — what each covers, the order they are best read +in, the terms they share. + +## 1. The three documents + +### [engine-audio-pipeline.md](engine-audio-pipeline.md) — Engine Audio Routing & Processing + +How one engine turns inputs into outputs. The proposal replaces today's +linked-list `AudioPipe` chain with a directed acyclic graph of processing +nodes, separates the engine into a network-aware I/O matrix layer and a pure +DSP graph layer with a clean lane boundary between them, and introduces +plugin/processing delay compensation (PDC) at the boundaries where it +matters. The proposal is contained to the inside of one engine; nothing in +this part touches the wire protocol, peer discovery, or cluster coordination. + +### [cluster-coordination.md](cluster-coordination.md) — Cluster Coordination & Transport + +How one or multiple engines, UIs, and IO boards hold together as a single coordinated state (or show so to say). The +proposal introduces the concept of a *cluster blueprint* — a single canonical desired-state +description that lives at an elected *coordinator* and is projected into +per-engine slices. It defines how structural edits and parameter edits are +routed differently (atomic transactions for the former, direct unicast for +the latter), how metering flows, how nodes are placed across engines, and +what happens when the coordinator dies. This part assumes the per-engine +model from the audio-graph proposal. + +### [state-coordination-protocol.md](state-coordination-protocol.md) — OAN State Coordination Protocol + +The wire-level protocol that gives the cluster proposal its atomic +reconfiguration guarantee. OSCP is a single-leader, broadcast-with- +correlation protocol that runs alongside the existing OAN audio, control, +mapping, and clock-sync streams on a single L2 segment. It provides +authority election, atomic transactional commits with sample-accurate apply +timestamps, late-joiner sync, and partition handling. This part is the +foundation that the cluster proposal stands on, but it is best read *after* +the cluster proposal — the cluster proposal motivates the protocol, the +protocol then explains the mechanism in detail. + +## 2. Reading order and why + +The three documents have a layered dependency that runs from the bottom up, +but they read better in roughly that order *as a story* rather than top-down: + +1. **[engine-audio-pipeline.md](engine-audio-pipeline.md) first.** It is the most self-contained of the + three. The DAG, the lanes, the I/O matrix split, the PDC — all stand on + their own without knowing anything about multi-engine clusters or about + how blueprints are distributed. +2. **[cluster-coordination.md](cluster-coordination.md) second.** Once the per-engine model is + in mind, the cluster questions make sense: how do nodes get placed across + engines, what does cross-engine wiring look like, where does cluster-wide + show state live. The cluster proposal references the audio-graph proposal + in several places (e.g. the per-engine RCU swap, the lane boundary). +3. **[state-coordination-protocol.md](state-coordination-protocol.md) last.** The cluster proposal names the property it + needs — "atomic cluster-wide reconfiguration at a sample-accurate + instant" — and OSCP is the + protocol that delivers it. + + +## 3. Shared glossary + +These are the terms that recur across all three documents. Each document also +has its own glossary for terms it introduces specifically; the entries below +are defined once here so the three docs do not have to redefine them. + +- **OAN** — OpenAudioNetwork. The raw-Ethernet audio transport submodule + (`OpenAudioNetwork/`) on which the entire OALS system runs. All + inter-process traffic in OALS — audio, control, mapping, clock sync, + cluster coordination — is carried over OAN EtherTypes; there is no IP-layer + transport anywhere in the system. New use cases get new EtherTypes. +- **EtherType** — the 16-bit value at offset 12 in an Ethernet frame that + identifies the upper-layer protocol. OAN currently uses 0x0681 (audio), + 0x0682 (discovery), 0x0683 (control), 0x0684 (clock sync); proposals + introduce 0x0685 (state coordination, OSCP) and 0x0686 (metering). +- **Peer / UID** — any device participating in an OAN segment. Each peer has + a 16-bit UID, unique on the segment, used as the address in OAN's + `LowLatHeader`. UID 0 is broadcast. +- **Engine** — a process that runs one audio graph plus its I/O matrix. The + authoritative owner of its own graph state and DSP. The current + implementation lives under `engine/`. +- **UI / control surface** — an operator-facing process (today: the Qt-based + `coreui`). Sends parameter edits direct to engines, sends structural edits + via the coordinator, subscribes to metering data from engines. +- **IO board / stage box** — a peer that contributes audio inputs and + outputs to the system but does no DSP. +- **Block** — one chunk of audio processed in one go. 64 mono float samples + (`AUDIO_DATA_SAMPLES_PER_PACKETS`). At 96 kHz that is ~667 µs of audio per + block; the whole engine ticks at block rate. +- **Cluster** — the set of engines, UIs and IO boards on one OAN segment, + working together on one show. +- **Coordinator** — the role that holds the cluster blueprint and projects + it into per-engine snapshots. Elected via OSCP. Never on the audio data + path. Defined in detail in the cluster-coordination proposal. +- **Blueprint** — the canonical cluster-wide desired-state description that + the coordinator holds and distributes. Carried as the opaque payload + inside OSCP transactions. +- **`apply_at_us`** — a sample-accurate future clock instant at which all + peers simultaneously apply a committed change. The mechanism that makes + cluster reconfiguration atomic. + +## 4. What is NOT in these proposals + +For clarity about scope: + +- **Show-file persistence (saved presets, scene memory, recall).** These + are application-layer concerns above OSCP. A UI may load a show file and + propose it as a blueprint, but the format and storage of show files are + not part of any of these three proposals. +- **Plugin authoring conventions** beyond the changes to the + engine-side `Node` interface implied by the audio-graph proposal. The + current plugin ABI under `plugins/loader/` will need migration; the + migration is mentioned where relevant but not designed in detail. +- **Remote operation across routed networks, web UIs, or any non-OAN + transport.** The proposals retain OALS's hard rule that all inter-process + transport is raw Ethernet via OAN. A future gateway peer could bridge + OAN to an IP-layer protocol, but no such gateway is part of these + proposals. +- **Auto-migration of nodes after engine death.** State recovery is out of + scope; operators recreate orphaned nodes manually. + +## 5. Status and how to read them + +All three documents are first-pass proposals. None has been reviewed; none +has been implemented. The decisions described in them are decisions I am +*recommending*, not decisions that are set in stone. Where significant +alternatives were considered, the alternatives and the reason for the +recommended choice are called out explicitly; smaller choices are explained +inline as proposal rationale. + +Each document carries its own "Open Questions" section listing items that +are genuinely unresolved. Comments, counter-proposals, and pushback on any +of it are the point of putting them out for review. diff --git a/proposals/cluster-coordination.md b/proposals/cluster-coordination.md new file mode 100644 index 0000000..b7e8f74 --- /dev/null +++ b/proposals/cluster-coordination.md @@ -0,0 +1,738 @@ +# Cluster Coordination & Transport + +**Status:** Proposal, not implemented, not yet reviewed +**Author:** Jonathan Reichardt +**Version:** Draft 0.1 +**Last updated:** 2026-06-01 +**Audience:** OALS contributors (current and future) +**Reading order:** second of three. Read [engine-audio-pipeline.md](engine-audio-pipeline.md) first; read [state-coordination-protocol.md](state-coordination-protocol.md) after. See [architecture-overview.md](architecture-overview.md) for the rationale. +**Prerequisites:** the per-engine model proposed in [engine-audio-pipeline.md](engine-audio-pipeline.md) (DAG of nodes, lane boundary, RCU graph swap). Familiarity with OAN at the level of `common/packet_structs.h` and `netutils/LowLatSocket.h`. + +## Abstract + +This document proposes how multiple engines, UIs, and IO boards on one OAN +segment hold together as a single show. It introduces the **cluster +blueprint** — one canonical, coordinator-held desired-state description of +everything in the cluster, projected into per-engine snapshots — and separates +the two kinds of edits that operators make: parameter edits (a fader move, an +EQ tweak) which go UI→engine direct, and structural edits (add a node, rewire) +which go through OSCP as atomic transactions with a sample-accurate apply +instant. It also covers how nodes are placed across engines, how metering data +flows from engines to UIs, and how the cluster behaves when the coordinator +dies. The per-engine model from [engine-audio-pipeline.md](engine-audio-pipeline.md) is assumed; the OSCP +protocol that delivers the atomic-commit primitive is described in +[state-coordination-protocol.md](state-coordination-protocol.md). + +## 0. Glossary + +Terms specific to this document. Cluster-wide terms shared with the other two +proposals — OAN, EtherType, Peer/UID, Engine, UI, IO board, Block, Cluster, +Coordinator, Blueprint, `apply_at_us` — are defined in [architecture-overview.md §3](architecture-overview.md#3-shared-glossary). + +- **OSCP** — OAN State Coordination Protocol. The proposed transactional + coordination protocol that handles authority election, atomic commits with + sample-accurate apply timestamps, late-joiner sync, and partition handling. + Runs on its own EtherType (`ETH_PROTO_OANSTATE`, 0x0685). Defined in + [state-coordination-protocol.md](state-coordination-protocol.md). +- **Authority** — the OSCP-elected peer responsible for serialising and + committing changes. In OALS, the coordinator role is fused with the OSCP + authority role: whichever device the OSCP election picks is the coordinator. +- **Snapshot (per-engine)** — the declarative description of one engine's + slice of the blueprint, sent to that engine after each commit. The engine + reconciles, compiles, and RCU-swaps to match. +- **Project / projection** — the coordinator's transformation from blueprint + to per-engine snapshots. Cross-engine edges are materialised into + egress/ingress nodes plus I/O matrix routes during projection. +- **Placement** — which engine a given node runs on. A stable property of the + node, decided at creation, persisted in the blueprint. +- **Domain** — an operator-level organisational container (e.g. FOH domain, MON + domain) holding defaults and metadata. New nodes created in a domain + inherit its defaults (preferred engine, tags, naming conventions). No + runtime audio effect — engines do not know that domains exist. +- **Path co-location** — the principle that an audio processing path should + live on one engine end-to-end where possible. Cross-engine paths are an + explicit advanced choice with the limitations described in [§4.3](#43-limitations-of-cross-engine-paths). +- **Param edit** — a runtime change to a single parameter (fader, threshold, + EQ frequency). Latency-critical, lossy-tolerant. Bypasses OSCP, goes + UI→engine direct on `OANCONTROL`. +- **Structural edit** — a change to graph topology or metadata (add or + remove node, change an edge, rename, retag). Goes through OSCP as a + blueprint proposal. +- **Apply timer** — the per-engine timer armed by an OSCP COMMIT, which + fires at `apply_at_us` and triggers the engine's RCU graph swap. + +## 1. The problem and the principles + +In a OALS deployment there mightn not only be one engine. +There might be several engines sharing a +pool of DSP capacity, with one or more UIs (FOH, MON, sometimes more), IO +boards on stage and at FOH, all on one OAN segment. The questions this +document answers: + +- Where does the cluster-wide show state live, and what happens to it when + something dies? +- How does a UI change something, and how does that propagate to the engines + that need to know? +- How do nodes get placed across engines, and what happens when a path being + built runs out of room on its current engine? +- How are metering values transported back to UIs? +- What transport carries each of these? + +Rather than enumerate concrete decisions up front, this proposal works from a +set of per-role principles. Most of the concrete behaviour described in later +sections falls out of these. + +**Engine.** +- Owns audio. Authoritative for the state of its own graph and DSP. +- Knows: one graph, its I/O matrix, its metering taps, its own identity. +- Doesn't know: other engines, operators, domains, ownership tags, show metadata. +- Talks to: the coordinator (declarative snapshots, ack/error), UIs sending param edits + (direct), UIs subscribed to its metering (direct). +- Never blocks audio on coordinator state. If the coordinator is dead, the engine keeps + running its last committed blueprint slice forever. + +**Node.** +- Lives on exactly one engine. Always. +- Stable identity (assigned by proposer, canonical once OSCP-committed), stable placement. +- DSP state (filter history, envelopes, delay-line contents) is local to its engine. Moving + a node between engines means destroying and recreating it — its state is lost. + +**Path.** +- A connected sequence of nodes should live on one engine end-to-end by + default. +- Cross-engine "wiring" is an egress node on the source engine plus an + ingress node on the destination engine — a send over the wire. PDC and + latency groups do not reach across this boundary (see [§4.3](#43-limitations-of-cross-engine-paths)). +- The coordinator places new nodes near their neighbours by default. + +**Coordinator.** +- Owns the cluster blueprint and projects it into per-engine snapshots. +- Owns show metadata (domain defaults, names, scenes, tags, per-node revisions). +- Elected via OSCP as the OSCP authority. Lives on whichever device the election picked. +- Never on the data path (audio, metering values). Only on the control path + (subscriptions, snapshots, ack/errors, structural edits). +- Stable to engine failure: if the coordinator whilst being on an engine and it dies, a new authority is elected + and re-issues snapshots; other engines keep running their last committed slices through + the transition. + +**UI.** +- Renders cluster blueprint state, sends edits, subscribes to metering streams. + (How it actually does this - node graph, fixed channels and busses, etc. is stll to be determined) +- Talks to the coordinator for structural edits and discovery; talks to engines directly + for param edits and metering data. +- Authoritative for nothing. Holds local caches for responsiveness, but truth is at the + coordinator (for structure) or the engine (for runtime params). + +**Domain.** +- A UX / coordinator-level container. Holds defaults and metadata. +- No runtime audio effect. Engines don't know domains exist. +- Multiple domains coexist; nodes from different domains can be wired together, no system + constraint against it. + +A few derived rules follow from the above: + +- Audio never depends on UI presence. +- Audio never depends on coordinator presence. (Audio depends on the + *blueprint* having been committed at some point; while running, audio is + independent of the coordinator.) +- There is exactly one logical graph view of the cluster — the blueprint. + There is no parallel "FOH graph" vs. "MON graph" at the system level; + those are conventions implemented as tags plus UI filters. +- The engine knows nothing about operators or domains. + +## 2. The transport stack + +The proposal preserves OALS's hard rule: **all inter-process transport is +raw Ethernet via OAN.** No TCP, UDP, WebSocket, gRPC, HTTP. New use cases +get new EtherTypes alongside the existing ones. This is a deliberate +architectural constraint; the consequences are discussed below. + +| EtherType | Name | Purpose | +|---|---|---| +| `0x0681` | `OANAUDIO` | Audio data (existing) | +| `0x0682` | `OANDISCO` | Mapping / discovery (existing) | +| `0x0683` | `OANCONTROL` | Per-element parameter updates: faders, thresholds, etc. (existing, evolved) | +| `0x0684` | `OANSYNC` | Clock sync (existing) | +| `0x0685` | `OANSTATE` (OSCP) | Cluster blueprint coordination — election, proposal, atomic commit (proposed in `state-coordination-protocol.md`) | +| `0x0686` | `OANMETER` | Engine → UI metering data, fire-and-forget | + +Compared to today the proposal adds **two new EtherTypes, `OANSTATE` and `OANMETER`**. + +### What raw-Ethernet-only buys, and what it costs + +Buys: one transport for the whole system; consistent reachability semantics +(a peer that can hear another's audio can configure it); no TCP-stack +dependencies; no need for IP-layer routing anywhere in the system. Engines +and UIs all use the same `LowLatSocket` machinery already built for audio. + +Costs: every UI must sit on the OAN segment, or be bridged to it at L2. A +tablet on Wi-Fi works if the access point bridges its clients onto the wired +LAN carrying OAN (the normal case). A browser UI on an arbitrary routed +network is not directly possible. Remote operation across the public internet is not +possible without an explicit gateway peer that bridges OAN to some IP +protocol — and no such gateway is part of this proposal. + +This constraint is a deliberate choice. The path to web UIs or remote +operation, if those are ever wanted, is a single dedicated gateway peer +rather than opening up the whole transport stack. + +## 3. OSCP — the coordination protocol + +OSCP is described in full in [state-coordination-protocol.md](state-coordination-protocol.md). This section summarises what +OSCP provides to the cluster proposal and calls out OALS-specific overrides +and dependencies. + +### 3.1 What OSCP provides + +- **Single-leader-per-partition authority election** with a BMCA-style + priority ladder. +- **Atomic transactional commits** with sample-accurate `apply_at_us` + timestamps. PROPOSE → PREPARE → COMMIT/ABORT. Every peer applies the new + state at the same clock instant. +- **Late-joiner sync** via `OSCP_STATE_REQUEST` / `OSCP_STATE`. +- **Partition-tolerant heal** with deterministic election-based winner + selection — no CRDT merging; explicit "your changes were discarded" UX + via `heal_info`. +- **Session-only state** — every boot starts blank. Eliminates the + stale-config-from-last-gig failure mode. +- **Opaque payload** — OSCP does not define what is in the bytes. The + application layer (OALS) does. + +### 3.2 OALS overrides and dependencies on OSCP + +**Payload size.** A realistic OALS blueprint (dozens of nodes, edges, +metadata) substantially exceeds one Ethernet frame. A rough estimate for +a 64-channel show with five-element strips lands in the 20–50 KB range, +or 15–30× the standard 1500-byte Ethernet MTU. The cluster proposal +therefore relies on OSCP's in-protocol fragmentation ([state-coordination-protocol.md](state-coordination-protocol.md) +§6.4): the coordinator sends a sequence of fragments tagged with one +transaction id, receivers reassemble before applying, and atomicity is +preserved. Application-layer chunking (splitting a blueprint update +into multiple OSCP transactions) would lose the atomicity guarantee, +which is the central reason for using OSCP at all, so the fragmentation +belongs inside the protocol — not in OALS. + +The relevant fragment-size constant: fragments are sized to **the +standard 1500-byte MTU minus headers**. No jumbo-frame assumption — see +[§3.4](#34-mtu-and-jumbo-frames) of this document, and [state-coordination-protocol.md §6.4](state-coordination-protocol.md#64-in-protocol-fragmentation) for the protocol-side +rationale. + +### 3.3 The boot story + +OSCP's session-only state has a sharp consequence: **a freshly powered +OALS network has no audio until a UI joins and proposes a blueprint.** +Engines come up on the OSCP `UNCONFIGURED` → `LISTENING` → ACTIVE-with- +empty-state path, with no graph loaded and no audio flowing. + +This is by design — it eliminates the stale-config-from-last-gig failure +mode. For a tour rig where an operator brings up a UI as part of setup, +it is the right behaviour. For fixed-install scenarios that want audio on +power-up before any UI presence, the right answer is a small auto-loader +peer that proposes a saved configuration on boot. No such peer is part +of this proposal yet. + +### 3.4 MTU and jumbo frames + +The proposal assumes **standard Ethernet MTU (1500 bytes)** everywhere. +Every frame on the wire — audio bundles, control packets, OSCP fragments +— is sized to fit a 1500-byte L2 payload. + +Jumbo frames (typically 9000-byte MTU) are attractive for OSCP +specifically because they would let a blueprint distribute in fewer +fragments. They are a deployment-environment optimisation rather than a +portable assumption, for three reasons: + +- Every NIC, switch, and bridge on the L2 path must be configured for the + chosen MTU. Oversized frames are dropped — often silently — by + equipment that does not expect them. +- Many deployment environments break this assumption: consumer NICs + defaulting to 1500, Wi-Fi access points (the wireless side rarely + supports jumbos), unmanaged switches, USB-Ethernet adapters. +- Even on jumbo-capable hardware, larger frames trade head-of-line + blocking against overhead savings — a 9000-byte frame takes ~72 µs to + serialise at gigabit Ethernet, vs. ~12 µs for a 1500-byte frame. On the + audio path that is a substantial fraction of the 667 µs block budget. + +The proposal therefore ignores jumbo frames. **Both OSCP fragmentation +and the audio path stay at standard frames.** A future extension can +add per-peer MTU advertisement in `MappingData`, let the authority size +OSCP fragments to the cluster-minimum MTU, and optionally allow audio +bundles to grow on segments where every peer supports it. None of that +is needed for the proposal to land, and deferring it keeps the base case +predictable across deployments. + +## 4. The cluster blueprint + +The blueprint is the payload OSCP carries: the canonical desired state of +everything across the cluster. + +### 4.1 What is in it + +- **Nodes.** Every node across the cluster, identified by a stable + `node_id`. Each carries: type, placement (`engine_uid`), port topology, + parameters (the values, not just the schema), tags, revision counter. +- **Edges.** Every connection between nodes — intra-engine or + cross-engine. Carries `(src_node_id, src_port)` → `(dst_node_id, + dst_port)`. +- **Cluster-wide metadata.** Domain definitions and their defaults. Named + latency groups and their members. Channel and lane naming. Other + show-shaped metadata that does not fit inside a single node. +- **Per-engine I/O matrix configuration.** Source-to-input-lane routing + tables and output-lane-to-destination routing tables. + +That is the complete picture. Parameter values *are* in the blueprint — +[§5](#5-edit-routing) covers the implications of that for edit volume. + +### 4.2 Projection: blueprint to per-engine snapshots + +When the coordinator commits a new blueprint via OSCP, it projects the +blueprint into one snapshot per engine. For engine E, the snapshot +contains: + +- All nodes with `placement = E`. +- All intra-E edges (both endpoints on E). +- For cross-engine edges that touch E: the materialisation as I/O matrix + entries. + - If E is the source side of the cross-engine edge: an egress sink node + writing to an output lane, plus an I/O matrix entry that sends that + lane to the destination engine. + - If E is the destination side: a source node reading from an input + lane, plus an I/O matrix entry receiving from the source engine on + that lane. +- The latency groups that include nodes on E, so PDC's group-equalisation + pass can run per-engine on members that are E-local — see [§4.3](#43-limitations-of-cross-engine-paths) for the + limitation this carries. + +The engine receiving its snapshot does not need to know that any of this +came from cross-engine wiring. It sees a graph with some source/sink +nodes wired to its I/O matrix. The cross-engine bookkeeping is entirely +the coordinator's job. + +### 4.3 Limitations of cross-engine paths + +Two things do not work across engine boundaries; the operator must be +informed of this: + +- **PDC does not span engines.** Each engine's PDC pass operates on its + own graph; cross-engine egress/ingress is opaque to it. A + latency-aware design therefore keeps PDC-relevant nodes co-located. +- **Latency groups do not equalise across engines.** Same reason. Group + members on different engines cannot be auto-aligned. + +These limitations are the reason for the path co-location principle in +[§1](#1-the-problem-and-the-principles); they are not bugs of the proposal but consequences of the data-path +boundary. + +## 5. Edit routing + +The proposal splits operator edits into two classes that travel on two +different transports. + +### 5.1 Parameter edits — direct, fast + +A parameter edit changes the value of one parameter on one node (a fader +move, a compressor threshold tweak, an EQ frequency change). It carries: + +- The target node, the parameter id within that node, the new value. +- The expected per-node revision counter, for concurrent-edit safety. + +The UI sends a `ControlPacket` on `OANCONTROL` directly to the engine +that owns the node. The engine checks the revision, applies the change if +the revision matches, bumps the revision counter, and returns an ack or +nack. + +**The coordinator is not in the loop for parameter edits.** It learns +about them asynchronously — either by the engine reporting its current +parameter state in an OSCP `STATE_REQUEST` response when asked, or via a +periodic engine→coordinator state-sync mechanism whose exact form is left +as an open question ([§9](#9-open-questions)). The principle is that parameter state must +survive an authority change without each fader having to be re-asked. + +Parameter edits are *idempotent at the revision level*: a UI that does +not see its revision confirmed resends. The engine drops duplicates, +since the same revision target is already applied. + +This path is unaffected by OSCP transactions, authority elections, or +any coordinator state. **A coordinator-less interval has no impact on +parameter editing.** + +### 5.2 Structural edits — atomic, slow + +Structural edits change the *shape* of the cluster: add or remove a +node, change an edge, rename, retag, change placement. They go through +OSCP. + +A UI proposes the new blueprint via `OSCP_PROPOSE` to the current authority. The +authority validates, runs PREPARE, aggregates ACKs, and either COMMITs — +with an `apply_at_us` 50–100 ms in the future — or ABORTs. On COMMIT, +every engine arms its apply timer; at `apply_at_us`, every engine +RCU-swaps its slice simultaneously. + +Structural-edit traffic is **dramatically lower-volume than parameter +edits.** An operator adds a node every several seconds at peak; OSCP +transactions happen at human pace. + +### 5.3 Why this split + +- Parameter edits are latency-critical (a fader move should land in ≤ a + few milliseconds) and loss-tolerant (a missed move is corrected by the + next one). They are exactly the wrong fit for OSCP's transactional + machinery, which buys atomicity at the cost of ~150–300 ms per + transaction. +- Structural edits are atomicity-critical (a cluster with half-applied + wiring is wrong) and latency-tolerant (waiting 150 ms to commit a node + insertion is fine). They are exactly the right fit for OSCP. + +The split is also reflected in the OSCP draft itself: [§1.2](state-coordination-protocol.md#12-out-of-scope) explicitly +excludes per-element parameter updates from OSCP's scope, so the two +proposals are aligned. + +> **Alternatives considered — everything through OSCP.** The simpler +> design routes every edit, parameter and structural alike, through OSCP +> transactions. It has one real advantage: parameter state lives in the +> blueprint automatically, eliminating [§9](#9-open-questions)'s open question about +> parameter-state survival across authority change. The proposal rejects +> it for two reasons: (1) operator latency. A fader move that has to +> wait for PROPOSE → PREPARE → COMMIT → apply ≈ 150–300 ms on a healthy +> LAN is not acceptable for a console; +> (2) Coordinator-failure brittleness. A coordinator outage would freeze every +> fader on every UI for the duration of the re-election (seconds at worst). +> The split here keeps parameter editing fully independent of OSCP state, +> at the cost of needing to recover parameter state separately after an authority +> change. + +## 6. Metering + +Metering has two halves: the *subscription* (which UI wants which taps) and +the *data flow* (the actual meter values). They have very different +characteristics and are described separately. + +### 6.1 Subscription + +A UI subscribes to a tap by telling the owning engine "send node X's +metering output Y at 30 Hz, addressed to my UID." The engine starts +publishing meter frames for that tap. Unsubscription is symmetric. + +Subscription discovery — knowing which engine owns which node — comes +from the cluster blueprint, which the UI watches via OSCP. The UI +therefore knows the `(engine_uid, node_id)` tuple for every node and +sends the subscribe message directly to the owning engine on +`OANCONTROL`. + +Subscription uses the parameter-edit-style transport: low-volume, +idempotent (re-sending the same subscribe is a no-op), revision-checked. +It is *not* part of the blueprint — metering is a runtime concern, not a +structural one. + +### 6.2 Data flow + +Engines publish meter frames on `OANMETER` (the new EtherType). Each +frame is self-contained: tap id, value(s), timestamp. Fire-and-forget; +no acks, no retransmits. A missed frame is gone, and the next one +arrives in 16–33 ms. + +Addressing is unicast to the subscribing UI's UID, using OAN's existing +UID-based addressing in `LowLatHeader`. Multiple UIs subscribing to the +same tap each receive their own unicast frame. + +Where one UI subscribes to many taps on the same engine, the publisher +batches them into a single multi-tap `OANMETER` frame per update tick +rather than sending one frame per subscription. This keeps per-frame +overhead bounded as subscription counts grow; the exact frame layout is +deferred ([§9](#9-open-questions)). + +### 6.3 Node-intrinsic vs. tap-point metering + +The proposal distinguishes two kinds of metering: + +- **Node-intrinsic** — measurements only the node can produce (a + compressor's gain reduction, an EQ band's activity). Published by the + node from inside its `process()` via the standard metering interface. +- **Tap-point I/O metering** — input and output levels. Almost every + audible node opts in to producing these via a shared mixin (or a + compiler-inserted side-tap), so the operator can click a node and see + I/O level, gain reduction, and any node-intrinsic values without + explicit wiring. Utility nodes (`DelayNode` and similar) opt out. + +Both kinds use the same transport (`OANMETER`) and the same subscription +mechanism. The distinction is internal to the node interface and not +visible to the UI. + +A separate `MeterNode` node type exists for the cases the per-node model +does not cover — metering at an arbitrary point on an edge, or +specialised meter types (LUFS, true-peak, spectrum analyser). It is a +node that performs only metering. + +The transport mechanism inside the engine is a lock-free slot per +metering output. The node writes its measurement every block (or every N +blocks for 30–60 Hz update rates) via a single atomic store. A non-RT +publisher thread reads currently-subscribed slots, packs them into +frames, and sends. The audio thread is never aware of subscriptions or +UIs. + +## 7. Placement + +Nodes live on engines. Which engine they live on is decided when the +node is created and is sticky: recompiles do not re-place, and only +explicit operator action or engine-death recovery moves a node. + +### 7.1 Placement decision rule + +Applied in order when a new node is being added: + +1. **Explicit pin.** If the operator pinned the node (or the node type + requires a specific engine capability — e.g. a VST-wrapper node on + engines that advertise VST support), place there. Failure (engine + dead or no capacity) is reported to the operator as an error; there + is no silent fallback. +2. **Path co-location.** If the node has neighbours in an existing path, + prefer their engine. +3. **Domain preferred engine.** If the node is created within a domain + that has a preferred engine, try that. +4. **Domain pool restriction.** A hard filter that narrows the eligible + engines. If a domain restricts itself to engines {3, 4, 5}, the + previous rules cannot pick engine 6. +5. **Domain placement policy.** Among the remaining eligible engines, + pick according to the domain's configured policy: + - **Pack** (default for mix-style domains such as FOH and MON): fill + the engine that already holds nodes from this domain up to a + configurable capacity threshold before moving on to the next. The + intent is that channels created early stay clustered, so the buses + and sends that connect them later land on the same engine and stay + intra-engine. + - **Spread**: pick the least-loaded eligible engine. Suitable for + domains where there is no expected "audio-flow community" between + newly-created nodes — e.g. a recording-bus or measurement domain + where every channel is independent of the others. + + The pack threshold and the default policy per domain are operator-set. + +Capacity overflow during single-node addition never silently splits a +path; it prompts the operator with concrete options ([§7.3](#73-capacity-overflow-during-single-node-addition)). Bulk +reorganisation of placement after the fact is handled by an explicit +*rebalance* action ([§7.4](#74-rebalancing)) rather than by re-running the rule above +across the existing graph. + +### 7.2 Engine capabilities + +Engines advertise capabilities — VST runtime present, hardware-bound +I/O, and so on — in `MappingPacket`. Nodes with capability requirements +(a node type whose plugin only loads on engines with capability X) are +implicitly pinned to the eligible engine set; the coordinator filters +before applying the placement rules above. + +This mechanism is not required for the proposal's core flow, but the +placement rule chain is general enough to accommodate it without +change. + +### 7.3 Capacity overflow during single-node addition + +When the rule in [§7.1](#71-placement-decision-rule) can find no eligible engine with capacity for a +new node, the coordinator prompts the operator with two options: + +- **Cross-engine link.** Place the new node on an eligible engine that + does have capacity, and introduce the cross-engine egress/ingress + pair on the connecting edges, with the limitations described in + [§4.3](#43-limitations-of-cross-engine-paths) (no PDC across the boundary, no latency-group equalisation + across it). +- **Pin elsewhere.** Operator picks the destination engine explicitly, + overriding the domain pool restriction if they choose. + +No automatic "move the existing path to a different engine" option is +offered at this point. The proposal does not have a sane way to decide, +at single-node-addition time, what counts as "the path" or which +neighbours are safe to move (existing nodes may be mid-show with active +audio and DSP state). Operator-driven bulk reorganisation is the +*rebalance* action ([§7.4](#74-rebalancing)), not a heuristic embedded in the overflow +dialog. + +### 7.4 Rebalancing + +The greedy per-node placement rule in [§7.1](#71-placement-decision-rule) is good enough for the live +case (one operator action adds one node) but is not expected to produce +a clean global layout when the operator builds out a substantial chunk +of the show in one sitting. After channels are created, sends are +wired, buses are added, the result will typically contain cross-engine +edges that exist only because the placement rule did not know what was +coming. + +To address this, the coordinator exposes a **Rebalance** action. When +the operator invokes it, the coordinator runs a global placement pass +over the current blueprint that tries to find better cut lines between +engines than the greedy history happened to produce. The pass is +parameterised by an **aggressiveness level** that controls what kinds +of edges are allowed to be cut: + +1. **Conservative.** No cuts at all; place everything on one engine if + it fits. Fails up to the next level if it does not. +2. **Latency-group-respecting.** Cuts permitted between any two nodes + *except* members of the same latency group. Within-group nodes + always co-locate. +3. **Branch-respecting.** Cuts further restricted to low-fan-in / + low-fan-out points — the "natural" branch borders. A send tap from a + channel to a monitor bus is a fine cut point; mid-channel-strip is + not. +4. **No mercy.** Any edge may be cut. Used when nothing else fits. + +The coordinator attempts the highest level the operator selected and +escalates only if no feasible placement exists at that level. The +operator sees a summary of what the pass actually did: which level it +ran at, how many nodes moved, how many cross-engine edges were +inserted, and which latency groups were preserved or violated. + +The cost of rebalancing is real: moving a node between engines destroys +that node's DSP state (filter histories, envelope followers, delay-line +contents). Rebalance is therefore a setup-time / soundcheck-time +action; the coordinator warns the operator if the action would affect +nodes currently passing audio, and the operator must confirm. Live-show +rebalancing during a song is not a supported use case. + +The rebalance pass is the global counterpart to [§7.1](#71-placement-decision-rule)'s greedy local +placement. Neither is sufficient alone: the local rule is needed for +live additions where there is no time and no operator attention for a +global re-layout, and the rebalance action is needed because no +greedy-local rule can be expected to produce a clean global layout in +the face of the order in which an operator happens to build a show. + +### 7.5 Placement is operator-visible + +Each node in the UI shows which engine it lives on (indicator, +colour-coding, or similar). Cross-engine edges are rendered as visually +distinct. The operator can see at a glance whether a path is +single-engine or split — preventing the failure mode where two paths +look identical but one of them crosses an engine boundary. + +### 7.6 Engine death + +If an engine dies, the nodes that lived on it are gone. The rest of the +show keeps playing: the other engines are unaffected, since each one's +last-committed slice keeps running indefinitely and audio does not +depend on the coordinator after commit. The operator can recreate or +re-pin the lost nodes on surviving engines manually. + +Auto-migration of nodes from a dead engine is intentionally out of +scope. + +## 8. How it all flows — three example walk-throughs + +### 8.1 Fresh boot to first audio + +1. Engines and IO boards power up. Each enters OSCP `UNCONFIGURED` → + `LISTENING`. +2. No authority exists yet. After `T_listen`, one devce (per OSCP [§7.4](state-coordination-protocol.md#74-election-state-machine)) + wins the contested election and becomes ACTIVE-authority with empty + state, generation = 1, version = 0. +3. Other devices become ACTIVE-slaves of that authority. +4. No graph is loaded on any engine; no audio flows. +5. The operator brings up the FOH UI. It joins OSCP, finds the + authority, and learns the (empty) current blueprint. +6. The operator loads a saved show file on the UI. The UI proposes the + loaded blueprint via `OSCP_PROPOSE`. +7. The authority validates, PREPAREs, aggregates ACKs from all engines, + and COMMITs with `apply_at_us = now + 80 ms`. +8. At `apply_at_us`, every engine simultaneously swaps its compiled + graph in. Audio starts flowing. + +### 8.2 Adding a node mid-show + +1. The operator clicks "add compressor" on a channel in the FOH UI. +2. The UI assigns a fresh `node_id` for the new compressor and locally + constructs the new blueprint (current blueprint + the new node + the + new edges into and out of it). Placement is decided: the + compressor's neighbours all live on engine 2, so the compressor goes + on engine 2 by path co-location. +3. The UI sends `OSCP_PROPOSE` to the current authority. +4. The authority validates and broadcasts PREPARE to all engines. +5. Each engine validates the slice that affects it: engine 2 sees a new + node plus new edges on its local graph; other engines see no change. + All ACK OK. +6. The authority broadcasts COMMIT with `apply_at_us`. +7. At `apply_at_us`, engine 2 RCU-swaps its compiled graph to the new + version with the compressor inserted. Other engines RCU-swap their + slices too (a no-op for them). The compressor's inputs and outputs + are seamlessly wired because the audio was already flowing through + the neighbouring nodes; the new node just inserts. + +### 8.3 Authority dies, audio keeps flowing + +1. Engine 3 was the OSCP authority. It physically loses power. +2. Other engines stop seeing `OSCP_ANNOUNCE` from engine 3. After + `T_announce_timeout × N`, surviving engines re-enter ASPIRANT and + re-elect. +3. Engine 1 wins. It bumps `generation`, issues `OSCP_STATE_REQUEST` to + the surviving peers to discover the most recent committed state, + adopts it, and starts broadcasting ANNOUNCE. +4. Throughout this election (about ten seconds in the worst case), + **every surviving engine has kept running its last-committed slice + continuously.** Audio does not pause. Parameter edits continue + working, since UIs still talk directly to engines on `OANCONTROL`. +5. UIs detect the authority change via the new ANNOUNCE (different + `authority_uid`). Any in-flight structural-edit proposals from + before the election are implicitly aborted; UIs see the timeout and + resubmit to the new authority if the edit is still relevant. +6. The nodes that lived on engine 3 are gone. Their slots in the + blueprint remain (the new authority adopted the same blueprint + state), but no engine is running them. The operator sees those + nodes as "orphaned" in the UI and can recreate them on surviving + engines. + +## 9. Open questions + +1. **Parameter-state survival across authority change.** OSCP [§1.2](state-coordination-protocol.md#12-out-of-scope) + explicitly excludes parameter edits from OSCP, so engines are the + authoritative holders of parameter state. When the authority + changes, the new authority needs to learn current parameter values + from somewhere — it does not have them. Three candidate mechanisms: + (a) a periodic engine→coordinator parameter-state-sync at low rate, + (b) the authority change triggering a one-shot engine→authority + parameter dump, (c) UIs caching parameter state and re-pushing it on + authority change. + +2. **Metering frame layout.** [§6.2](#62-data-flow) commits to batching multiple taps + from one engine to one UI into a single `OANMETER` frame, but the + on-wire frame format (tap-id encoding, per-tap value width, optional + per-tap timestamp vs. one frame-level timestamp) is not yet + specified. + +3. **Subscription expiry and heartbeat.** If a UI dies without + unsubscribing, engines would keep publishing meter frames + indefinitely. A subscription TTL or heartbeat-based liveness check + is needed; the exact mechanism is unspecified. + +4. **Blueprint encoding.** The OSCP opaque payload is a byte blob and + OALS defines what is in it. The concrete schema is not specified + here and is deferred to a follow-up. + +5. **Domain conflict semantics.** Two operators in different domains + who happen to edit the same node (the system does not enforce + ownership) — what does the UI show? The likely answer is + last-writer-wins via the revision counter, with the second + operator's edit rejected as stale. Worth confirming explicitly. + +6. **Rebalance aggressiveness levels.** [§7.4](#74-rebalancing) proposes four levels + (conservative, latency-group-respecting, branch-respecting, no + mercy). The right number of levels and the exact definition of each + are open. In particular, "branch-respecting" relies on a definition + of "natural border" that has not been pinned down — fan-in/fan-out + thresholds, the role of explicit operator-tagged cut points, and + whether cross-domain edges should always be preferred cut points are + all candidates. Also open: whether rebalance should support partial + scope (rebalance one domain, or a selected set of nodes) rather than + always running over the whole blueprint. + +The need for **OSCP in-protocol fragmentation** is not listed here +because [§3.2](#32-oals-overrides-and-dependencies-on-oscp) resolves it as a required amendment to the OSCP draft +rather than an open question of this proposal. + +## 10. What is not in this document + +For clarity about scope: + +- **What happens inside one engine.** Covered by [engine-audio-pipeline.md](engine-audio-pipeline.md). +- **The OSCP protocol itself.** Covered by [state-coordination-protocol.md](state-coordination-protocol.md). +- **Clock synchronisation.** Covered separately by + `clock-sync-design.md`. +- **Persistence (show files, scenes).** An application-layer concern, + not a protocol-level one. OSCP treats persistence as out of scope; a + UI loads a show file and proposes the resulting blueprint to the + authority. diff --git a/proposals/engine-audio-pipeline.md b/proposals/engine-audio-pipeline.md new file mode 100644 index 0000000..873eb1c --- /dev/null +++ b/proposals/engine-audio-pipeline.md @@ -0,0 +1,634 @@ +# Engine Audio Routing & Processing — Design + +**Status:** Proposal, not implemented, not yet reviewed +**Author:** Jonathan Reichardt +**Version:** Draft 0.1 +**Last updated:** 2026-06-01 +**Audience:** OALS contributors +**Reading order:** see [architecture-overview.md](architecture-overview.md). This is the first of three proposal documents and can be read standalone. +**Prerequisites:** general familiarity with the OALS codebase, in particular the current `engine/` and `plugins/loader/AudioPipe.h`. + +## Abstract + +This document proposes a redesign of how audio is routed and processed *inside* +a single OALS engine, and of how the engine meets the network boundary. The +current implementation processes audio by walking a linked list of `AudioPipe` +objects on the realtime thread, with per-pipe mutexes and heap-backed queues on +the hot path; the only routing primitives are via the network, so local audio +often round-trips the wire. The proposal replaces the linked list with a +directed acyclic graph (DAG) of processing nodes, that is compiled off the audio thread into +a flat ordered execution list and atomically swapped in via Read-Copy-Update (RCU); separates the +engine into a network-aware I/O matrix layer and a pure-DSP graph layer +connected by a fixed lane boundary; and introduces processing delay +compensation that operates only on declared internal node latency, above that +boundary. + +--- + +## 0. Glossary + +Terms specific to this document. Cross-document terms (Block, OAN, peer, UID, engine, +etc.) are defined once in the shared glossary in [architecture-overview.md](architecture-overview.md). + +- **DAG** — Directed Acyclic Graph. A graph of nodes with directed connections and + no cycles (no loop back to where you started). The shape this proposal gives the + processing graph, because a DAG can be sorted into a clear "do A, then B, then C" + order. +- **Node** — one DSP unit in the graph (a filter, a comp, a summing bus, an input + source, an output sink). Has input ports, output ports, internal state, and a + `process()` function. +- **Port** — an input or output point on a node. Each port is a single mono signal. + A stereo node is just a node with two input ports and two output ports. +- **Edge** — a directed connection from one node's output port to another node's + input port. Carries one mono signal. +- **Lane** — a fixed mono signal slot at the boundary between the I/O matrix and the + graph. Input lanes are written by the I/O matrix and read by graph source nodes; + output lanes are written by graph sink nodes and read by the I/O matrix. +- **Edge buffer** — The way audio is actually passed from node to node - + a single mono `float[64]` slot from the engine-wide pool, assigned + by the compiler to one edge (or one lane). The producing node writes its output into + the slot, the consuming node reads from it, and the contents are then overwritten next + block. Edge buffers carry no state between blocks. +- **Node-internal state** — data owned by one node that persists across blocks: filter + histories, envelope-follower state, the ring buffer inside a `DelayNode`. Distinct + from edge buffers; the compiler does not touch it. +- **Topological sort (topo-sort)** — ordering the nodes of a DAG so that every node + comes after all its inputs. Gives you a flat list you can just walk in order. +- **I/O matrix** — the outer layer that owns everything network-shaped: jitter + buffering, packet bundling, mapping network sources to lanes and lanes to network + destinations. +- **Jitter buffer** — a ring of samples on the ingress side that absorbs the timing + variability of network packet arrival, so what the graph reads from a lane is + steady and gapless. +- **PDC** — Processing Delay Compensation. Automatically inserting delays so + that parallel paths through the graph stay sample-aligned at summing points and + at declared groups of outputs. Operates only on *internal* declared latency, + never on network jitter. +- **DelayNode** — a node whose only job is to delay its input by N samples, + inserted by the compiler during PDC. +- **Bus / Summing bus** — a node with K input ports and 1 output port that sums its + inputs. The only way to mix signals in this design. +- **Sidechain** — an input port on a node used for detection/control rather than + being summed into the main signal path (e.g. a comp's detection input). +- **RCU** — Read-Copy-Update. A lock-free pattern where readers see a fully-built + immutable structure via one atomic pointer load, and writers publish updates by + atomic pointer swap, with old versions reclaimed only after readers have moved + on. Used here to swap the compiled graph atomically without blocking the audio + thread. +- **SPSC ring** — Single-Producer, Single-Consumer ring buffer. Lock-free, + fixed-size, no allocation. Used wherever exactly one thread writes and one + thread reads. +- **BLOCKSIZE** - the size of a block of floats containing continuous audio samples, usually 64 +--- + +## 1. Background: how today's engine processes audio + +Audio in the current engine flows through a singly-linked list of `AudioPipe` objects. +A pipe processes a packet and calls `next->feed_packet`, or queues it for the updater +thread to drain later. The engine keeps a flat `array, 64>`, one +slot per channel, and the DSP thread scans all 64 every tick. + +Per-element packet handoff goes through one of two paths: `feed_packet` for direct, +synchronous delivery, and `push_packet` for queued delivery (drained later by the +updater thread). The queued path takes a `std::mutex` and pushes onto a +`std::queue` per pipe, per block (`AudioPipe.cpp:34`). The ingress +alignment path keeps each network source's samples in a `SampleStream` backed by a +`std::queue`. + +Routing between pipes is hardcoded into a small set of specialised pipe types: +`AudioInMtx` sums and aligns incoming network streams onto channels; `AudioSendMtx` +fans channels out to network destinations; `AudioDirectOut` sends a single channel to +a single destination. There is no general "connect this output to that input" +primitive — the only router is the network matrix. + +A "pipe chain" is a singly-linked list of these pipe types installed at one of the +engine's 64 channel slots. A pipe is identified by `(channel, index_in_chain)`. New +chains are constructed remotely by the UI sending a sequence of +`ControlPipeCreatePacket`s, accumulated and assembled by the engine. + +## 2. Motivation: why that needs to change + +The current design has five concrete problems that this proposal is designed to +address: + +- **The hot path allocates and locks.** `std::queue` is heap-backed and `push_packet` + takes a `std::mutex` per pipe, per block. The `SampleStream` is worse — it heap- + allocates per sample. This is unsafe in a realtime audio engine: heap allocations + have unbounded worst-case latency (they can block on the allocator's internal locks, + page in memory, trigger arena rebalancing), and `std::mutex` on a `SCHED_FIFO` + thread is a priority-inversion waiting to happen — if a lower-priority thread holds + the mutex, the audio thread blocks until the kernel scheduler sorts it out, which + can be milliseconds. Either one missing a deadline means dropouts. +- **Routing is hardcoded into magic pipe types.** Because "where does audio go" is + baked into `AudioInMtx`, `AudioSendMtx`, and `AudioDirectOut`, there is no general + routing primitive. Every new routing shape needs a new pipe type. +- **Local audio round-trips the network.** Because the processing unit *is* a network + `AudioPacket` and the only routing primitive is "send to a host", moving audio + between two chains on the same engine often goes back out over the wire and comes + back. +- **No real graph.** A chain is a straight line. No summing buses, no sidechains, no + parallel paths, no fan-out. +- **No latency compensation.** Parallel paths into a mix arrive misaligned and + comb-filter. + +Addressing these is a fairly fundamental redesign of the inner engine. The network +protocol, discovery, clock sync, and the UI's pipe-creation flow mostly survive; what +changes is *how an engine turns inputs into outputs*. + +--- + +## 3. Proposal, part 1: a DAG instead of a linked list + +Today's pipe chain is a straight line — element A processes a packet, hands it +to element B, which hands it to C. That works for "one input gets EQ'd and compressed and +goes out", but it is too rigid for anything more complex. A signal cannot be split +into two paths and merged again (parallel compression). A single bus cannot receive feeds +from multiple channels. A comp cannot have a sidechain input. The chain can only represent +the simplest possible processing topology. + +It's also opaque to the engine: a linked list of pointers tells the runtime nothing about +*what depends on what*. The DSP thread has to traverse the structure every block, +following pointers, locking mutexes to be safe, with no opportunity to know which work is +independent and which is sequential. Eventually parallelising it is essentially impossible. + +This proposal replaces the linked list with a **DAG — a directed acyclic graph** of +processing nodes. Nodes are the things that process audio (a filter, a comp, a summing +bus, a source, a sink). Edges are connections from one node's output to another node's +input. "Acyclic" means no loops back to where you started — feedback loops are forbidden +because they are algebraically unsolvable in one block of audio. + +A DAG can express anything a console needs: parallel paths, fan-out (one signal feeding +several places), summing buses, sidechains, group routing. And because it's acyclic, you +can **topologically sort** it — produce a flat list "do node A, then B, then C…" where +every node appears after all the nodes that feed it. The DSP thread's job becomes walking +that list once per block. No traversal, no pointer-chasing, no locking, no decisions about +ordering at runtime — the ordering was figured out once, off the audio thread, by whoever +compiled the graph. This is a very common approach in digital audio processing. + +That is the first half of the design: **describe what should happen as a DAG of nodes, +compile it to a flat ordered execution list, and run that list on the audio thread.** The +audio thread does no graph reasoning of any kind — it just runs the list. + +> **Alternatives considered.** The main alternative is to keep the current linked-list +> chain model and address its individual problems incrementally (replace the per-pipe +> mutex with a lock-free queue, add a separate "router" pipe type for fan-out, etc.). +> This was rejected because the chain model fundamentally cannot express the topologies +> a real console needs (parallel paths, sidechains, fan-in via summing) without bolting +> on enough special cases that the result is effectively a graph anyway — with worse +> ergonomics and no opportunity for the topological-sort optimisation that makes the +> hot path cheap. A second alternative — a fully cyclic graph with explicit feedback +> delay — is rejected as out of scope: audio feedback is rare in live-mixing use cases +> the system targets, and the implementation cost (per-cycle delay-line analysis, the +> sample-by-sample dependency tracking it requires) is not justified by the use case. + +## 4. Proposal, part 2: two layers separated by a lane boundary + +The second proposal: **stop using the network packet as the processing unit.** At present +`AudioPacket` does two jobs — it is the wire format (packed struct, `channel`, +`timestamp`) *and* the thing that flows through DSP. That conflation is the root cause of +half the problems above (local audio round-tripping the network, routing being hardcoded +into magic pipe types). + +This proposal splits the engine into two layers with a clean wall between them: + +``` + WIRE ──► [ I/O MATRIX ] ──► input lanes ──► [ THE GRAPH ] ──► output lanes ──► [ I/O MATRIX ] ──► WIRE + network concerns: pure DSP: network concerns: + jitter buffers, mono float[BLOCKSIZE] buffers, fan-out, + arrival alignment, a real node graph, timestamping, + packet bundling, compiled + run in order send + source→lane routing +``` + +*Figure 1: The two-layer split. The I/O matrix handles all network-shaped concerns; the +graph handles all DSP. Input and output lanes are the only handover format between them.* + +- The **I/O matrix** (outer layer) owns everything network-shaped: jitter buffering, + aligning packets that arrive at different times, packing/unpacking multi-channel + frames, and mapping network sources onto internal lanes. +- The **graph** (inner layer) — the DAG described in [§3](#3-proposal-part-1-a-dag-instead-of-a-linked-list) — is pure DSP on bare + mono `float[BLOCKSIZE]` buffers. It has no idea what's network and what's local. It just + reads "lane K", processes, writes "lane J". + +A clean handover format is needed between the two layers. The graph wants to do DSP on +plain aligned blocks of samples; the I/O matrix wants to do whatever it needs to do with +packets, timestamps, and bundling on the other side. The proposal introduces **lanes**: +fixed mono signal slots that act as the docking points between the two worlds. An input +lane is filled by the I/O matrix and read by some source node in the graph; an output +lane is written by some sink node in the graph and read by the I/O matrix. + +**A lane is the wall.** A sample sitting in an input lane is, by definition, already on +the local clock and gapless — the I/O matrix made it so. The graph treats lanes as +ground truth and never thinks about timestamps. A local source→sink path is now just two +nodes and an edge; it never touches the network. + +Everything below follows from this split. + +--- + +## 5. The graph (inner layer) + +### 5.1 Nodes, ports, lanes + +A **node** is one DSP unit. It has input ports and output ports, persistent state (filter +history, envelopes), and a `process()` that reads its inputs and writes its outputs. + +```cpp +struct ProcessContext { + const Block* const* ins; // ins[p], points at a const all-zero block if unconnected + Block* const* outs; // outs[p], node fills these + int num_in, num_out; + int nframes; // usually 64, but not hardcoded + uint64_t steady_time; // block start in samples since engine start + // room to grow: transport, flags, ... +}; + +class Node { +public: + virtual ~Node() = default; + virtual int num_inputs() const = 0; + virtual int num_outputs() const = 0; + virtual bool is_sidechain_input(int port) const { return false; } + virtual void process(const ProcessContext& ctx) = 0; // the hot path + virtual int latency_samples() const { return 0; } + virtual void apply_control(const ControlData& c) {} +}; +``` + +Notes on why it looks like this: + +- **`process` takes a context struct, not bare `float**`.** and passes an explicit frame count; + none hardcode block size. The payoff: partial blocks become possible (if ever necessary), sample-accurate + parameter ramps become possible, and the struct can grow later without re-touching + every node. The one discipline this requires: **loop to `ctx.nframes`, never to a + hardcoded 64.** +- **No `AudioPacket`, no channel, no timestamp inside a node.** Those are wire concerns + and they live in the I/O matrix only. +- **No per-sample virtual call.** The old `process_sample(float)` meant 64 virtual + dispatches per node per block. Nodes now get the whole block and loop internally — which + is also what lets them use SIMD. +- **No mutex, no queue in the base class.** For RT-safety they are gone/removed. + +The data carried between nodes lives in **edge buffers**: pre-allocated, 32-byte-aligned +mono `float[BLOCKSIZE]` slots drawn from one engine-wide pool. The pool is sized at graph +compile time ([§7.1](#71-the-compiler)); slots are never allocated on the audio thread. The compiler hands +out one edge buffer per edge — that is what the `Block*` pointers in `ProcessContext.ins` +and `ProcessContext.outs` actually point at. Each block of audio, the producing node +writes its output into the slot, the consuming node reads from it, and the slot's +contents are then meaningless until the next block overwrites them. Edge buffers carry +no state between blocks. + +Lanes are the same kind of object, just used at the layer boundary: input lanes are +edge buffers written by the I/O matrix and read by graph source nodes; output lanes are +edge buffers written by graph sink nodes and read by the I/O matrix. Nothing in the +audio path ever branches on "is this a lane or a wire" — it is all just `Block*`. + +Edge buffers are distinct from **node-internal state** (filter histories, envelope +followers, the ring buffer inside a `DelayNode`). Internal state is owned by the node +itself, persists across blocks, and is opaque to the compiler. The compiler manages +edge buffers; nodes manage their own state. The two never overlap. + +### 5.2 Summing is an explicit node + +When several outputs feed one place, something has to add them. This proposal makes that +a real node (`SummingBus`, K inputs → 1 output) rather than letting an input port silently +accept multiple edges. So **edges are always 1 output → 1 input**: fan-out is fine, +fan-in is forbidden — to mix signals, use a bus. + +This matches the mental model ("channels A, B, C go to group C"), keeps the edge rules +trivial, and gives latency compensation a clear anchor (the bus *is* the place where +alignment happens). + +### 5.3 Multi-channel formats (stereo, 5.1, LCR) without complicating the engine + +There is no inherent stereo concept today, and the system needs to handle arbitrary +formats. This proposal's approach is to **not** teach the audio primitives about channel +counts. Three separate ideas that the word "stereo" usually conflates: + +1. **The carrier is always mono.** A lane is one `float[BLOCKSIZE]`, always. A stereo bus is two + lanes. 5.1 is six. Buffers, summing, delay lines, SIMD, the matrix — all stay mono and + simple, forever. +2. **Format is metadata, and it lives in a layer above the engine.** "Lanes 4,5 are a + stereo pair (L,R)" is a tag. The audio path doesn't care about it — it just processes + mono lanes. That tag wants to live somewhere a UI (or several UIs, or other tools) can + see and agree on, without each one having to re-derive it. This is the **cluster + blueprint** held by the OSCP authority across the network — see the cluster-coordination + design doc. Format/grouping metadata, channel naming, fader-link groups, latency-group + definitions all live there, alongside the structural graph itself, and the authority + distributes them to engines atomically via OSCP. The engine's job stays narrow: process + mono lanes; the layer above decides what those lanes mean to operators. +3. **Channel-grouped behaviour lives inside node types.** A linked stereo comp is *one* + node with 2 in / 2 out that computes one gain reduction from both and applies it to + both. A panner is one node with 1 in / N out. The engine just sees "a node with some + ports" — the stereo-ness is internal to that node. + +So complexity goes where it is cheap: format-handling in the (off-the-hot-path) UI and +compiler, channel-grouped DSP inside the few nodes that want it. The engine core never +learns what stereo is. + +> **Alternatives considered.** The main alternative is to introduce typed multi-channel +> buffers — a `StereoBlock` distinct from `MonoBlock`, a `SurroundBlock`, etc. — and to +> have nodes declare their input/output formats statically. This was rejected because +> it pushes a non-trivial type system into the core engine for a problem that only +> exists at the operator-visible layer. Every primitive (SIMD operations, summing +> buses, delay lines, the buffer pool) would need format-aware variants; the compiler +> would need format-compatibility checks at every edge; format conversion between +> incompatible nodes (mono → stereo, stereo → 5.1) would need to be either auto- +> inserted or rejected. None of that complexity buys the user anything they cannot +> get from the proposal's "mono lanes + format-as-metadata" approach, in which a UI +> presents a stereo bus as one object backed by two lanes and a node-type author who +> wants linked-stereo behaviour just writes a node with 2 in / 2 out. + +### 5.4 Running the graph + +The compiler ([§7](#7-compiling-and-reconfiguring-the-graph)) turns the node set into a flat, topologically-sorted execution +list with all the buffer pointers pre-resolved. The DSP thread's entire job per block +becomes: + +```cpp +auto* g = current_graph.load(acquire); // one atomic load +for (int i = 0; i < g->order.size(); ++i) + g->order[i]->process(ctx_for(g, i)); // no locks, no allocs, no map lookups +block_counter.fetch_add(1, release); +``` + +Compare to today's 64-slot scan with a mutex + queue pop per node. This is both simpler +and dramatically cheaper. + +--- + +## 6. The I/O matrix (outer layer) + +This layer turns a stream of lossy, jittery, (in the future) possibly-bundled network packets into clean +lanes, and back. It's where today's `AudioInMtx` / `AudioSendMtx` / `AudioInPipe` / +`AudioDirectOut` / `SampleStream` all dissolve into one principled thing. + +### 6.1 Threading and the jitter buffer + +- **Ingress write** runs on the audiopoll thread: as packets arrive off the NIC, + unpack them and write samples into a **per-source SPSC jitter ring** — lock-free, + fixed-size, zero allocation. (This is what replaces `SampleStream`'s `std::queue`.) +- **Ingress read** runs on the DSP thread at the start of each block: pull exactly one + block of samples from each ring into the input lanes. After this, lanes are on-clock + and gapless. +- **Egress** runs on the DSP thread at block end: it writes output lanes into another SPSC + ring, and a **dedicated send thread** drains that ring and does the actual socket sends. + (Today `AudioDirectOut` sends straight from the DSP-adjacent path — a syscall on the hot + path. This moves it off.) + +### 6.2 Network arrival alignment (NOT latency compensation) + +The jitter ring reads from a point that sits `target_fill` samples behind the write head. +That fill depth absorbs network jitter. Per-stream arrival timing is measured +(`header.timestamp` vs local clock) — the same idea `AudioInMtx` does today, measuring the +spread and aligning streams — and each stream's fill depth is set accordingly. + +**This is deliberately separate from the graph's latency compensation, in concept and in +wording.** Arrival alignment is about the wire (jitter, loss, drift) and lives *below* the +lanes. Latency compensation ([§8](#8-latency-compensation)) is about declared internal processing delay and +lives *above* the lanes. The lane is the wall: by the time a sample is in a lane, arrival +alignment has already happened, invisibly, and the graph counts latency from zero. The +jitter buffer's fill depth is not allowed to leak upward as a latency number. + +Clock drift between a source and the local engine slowly fills or drains the ring. This +proposal handles it by skipping or inserting a sample when fill crosses a watermark — +cheap, and rare if clock sync is doing its job. A proper async sample-rate converter is +a separate component with its own design, out of scope here. + +### 6.3 Source → lane routing + +Mapping an incoming `(source_uid, source_channel)` to an input lane is an explicit table +(this replaces `AudioInPipe`'s implicit route-filter). It's part of the declarative graph +state ([§7.3](#73-engine-interface-declarative-reconciliation-not-deltas)), so it swaps atomically with everything else. A source can fan to +several lanes (one mic into two chains). + +### 6.4 Bundled multi-channel frames + +Today one `AudioData` frame carries one mono channel — at 96k that is a frame every +~667µs *per channel*, which is a lot of per-frame overhead at channel counts that matter. +This proposal adds a bundled frame that carries K channels under one header and one +timestamp: + +``` +CommonHeader | AudioBundleHeader{ channel_count K, flags } + | AudioBundleChannel{ source_channel, valid }[K] + | float[K][BLOCKSIZE] +``` + +*Figure 2: Wire layout of the proposed bundled multi-channel audio frame. K is fixed per +device. One `CommonHeader` (and one timestamp inside it) covers all K channels.* + +- **Fixed K per device** — predictable parsing and MTU budgeting. A per-channel `valid` + flag lets a sender skip data for silent channels. (~5–6 channels fit a 1500 MTU, ~34 a + jumbo frame.) +- **One timestamp for the whole bundle** — the K channels were captured together at one + clock instant, which is the whole point and makes ingress alignment land them all at the + same stream position. + +This lives entirely in the I/O matrix. The `AudioData` typedef and the Wireshark dissector +change; nothing in the graph notices. + +--- + +## 7. Compiling and reconfiguring the graph + +### 7.1 The compiler + +When the graph changes, a function on the **control thread** (off the audio thread, free to +allocate) turns the desired node+edge set into a ready-to-run `CompiledGraph`. Stages, in +order: + +1. **Reconcile — reuse vs. create.** This is what stops edits from clicking. Nodes are + identified by a stable id. An id that's new → create it; an id already live → **reuse the + existing node object** (so its filter/envelope state keeps running); an id that's gone → + retire it. The node registry is the stable thing; the compiled graph is just fresh wiring + over the same node objects. Moving a comp or inserting an EQ before it doesn't reset the + comp. +2. **Validate + check for cycles.** Edges must match real ports; no input may have more + than one incoming edge (that is what a bus is for); no cycles (this proposal rejects + cyclic graphs — see the alternatives note in [§3](#3-proposal-part-1-a-dag-instead-of-a-linked-list)). A failed compile is a **no-op** — + the running graph is untouched, nothing glitches, but an error is surfaced to the user. +3. **Topological sort** (Kahn's). Producers before consumers. Sidechain inputs count for + ordering even though they aren't summed. Ties broken by node id so the same graph always + compiles to the same order. +4. **Insert latency compensation** ([§8](#8-latency-compensation)) — runs here, before edge-buffer assignment, so + that any delay nodes inserted by the PDC pass appear as ordinary nodes in the graph + and have edge buffers allocated for them like any other. +5. **Assign edge buffers.** Allocate one mono `float[BLOCKSIZE]` slot from the engine-wide pool + per edge (and per lane), and record the slot id on the edge. The compiler may also + compute non-overlapping lifetimes — two edges whose lifetimes do not overlap can + share one slot, and a node whose single input has no other readers can write its + output back into the input's slot ("process in place"). These are pure memory-pool + optimisations of the per-edge slot assignment; they change neither the node API nor + the rest of the graph. The naive policy — one distinct slot per edge — costs a few + hundred kilobytes for a large show, which is acceptable; sharing is worth doing only + if a profile says so. **Edge-buffer assignment does not touch node-internal state + (filter histories, envelopes, delay-line contents): those belong to the node and are + carried across recompiles by the reconcile pass in stage 1.** +6. **Flatten** into per-node pointer slices so the hot loop does zero lookups. + +### 7.2 Swapping it in atomically (RCU) + +There is exactly one reader (the DSP thread) and one writer (the control thread), which +makes this the simplest possible RCU: + +- The control thread builds the whole new graph, then does one atomic pointer swap. +- The DSP thread loads that pointer once at the top of a block and uses it for the whole + block. So routing changes take effect cleanly from one BLOCKSIZE-long (64) sample block to the next — + the "atomic, sample-to-sample" requirement, at block granularity. +- **Nobody frees on the audio thread.** After a swap the old graph might still be in use for + the current block. The control thread hands the old graph to a **dedicated reclaimer thread**, + which waits until the DSP thread's block counter has advanced two blocks (safely past + any possible use, ~1.3ms) and then frees it. Freeing the graph frees the wiring and + buffer pool + but **not the nodes** — those are shared with the new graph and only destroyed when actually + removed (after the same wait). + +In the multi-engine setting, every engine's RCU swap is driven by the cluster coordinator's +OSCP commit (see the cluster-coordination design doc). When OSCP COMMITs a new blueprint with +an `apply_at_us` timestamp, each engine's control thread arms the swap for that clock instant. +At `apply_at_us`, every engine in the cluster swaps its pointer simultaneously — so what's +described above as per-engine block-aligned atomicity is, across the cluster, also a +clock-aligned simultaneous commit. The per-engine mechanics in this section are unchanged; +the trigger just comes from a coordinated apply timer instead of an immediate swap. + +### 7.3 Engine interface: declarative reconciliation, not deltas + +Upstream of each engine sits the **cluster coordinator** (an OSCP authority elected +across the engine pool; see `cluster-coordination.md`). The coordinator holds the +cluster blueprint and projects it into per-engine slices. Each engine receives its slice +as a complete declarative description of the nodes and edges it should be running, and +reconciles against what it currently has. Re-sending the same description is a no-op. + +This proposal uses a declarative model rather than a delta protocol ("add node", +"connect X to Y") specifically because the transport is lossy raw Ethernet with no +retransmit. A dropped delta is a permanent desync; a dropped full-state message just means +the engine keeps the last good graph until the next send. Idempotency is the reliability +mechanism — no acks, no resync handshake needed. (OSCP layers transactional guarantees on +top — PROPOSE / PREPARE / COMMIT — so the cluster-wide commit is atomic, but the +per-engine reconciliation logic described here is the same either way.) + +> **Alternatives considered.** The main alternative is an explicit delta/event protocol: +> the coordinator emits "add node N", "connect A → B", "remove node M" messages and the +> engine applies them in order. This is the model the current `ControlPipeCreate` packet +> sequence uses (one packet per element, accumulated by `seq`/`seq_max`). It was rejected +> for the new design because (a) the OAN transport is lossy raw Ethernet with no +> retransmit, so a dropped delta is a permanent desync requiring its own resync +> handshake, and (b) deltas couple the engine's behaviour to the coordinator's history, +> meaning a late-joining engine has to either replay the full event log or request a +> snapshot anyway — at which point the snapshot is doing the real work and the deltas +> are redundant. The declarative model collapses both cases into one mechanism. A second +> alternative — keeping the current `ControlPipeCreate` sequential format but extending +> it for graph topology — was rejected for the same reasons plus the added cost of +> maintaining two distinct shape-of-graph encodings during the transition. + +The one piece of machinery this needs: a full-state snapshot spans many packets. OSCP +handles the cross-engine framing and fragmentation; within each engine's slice a revision +is carried so the engine only compiles once the slice is complete. A half-received slice +is dropped, not applied. + +Node ids are assigned by the proposer (typically the UI that originated the edit) and are +stable across edits — that's what makes reconcile-by-id work. Once a proposal is OSCP- +committed, those ids become canonical across the cluster. Changing a node's port count +(e.g. mono comp → linked stereo) counts as a new node — its state resets, which is fine +because you explicitly changed the processing shape. + +--- + +## 8. Latency compensation + +Operates only on **declared internal node latency** (`latency_samples()`), above the lanes. The +lane is zero. (Again: this is not the jitter buffer — see 5.2.) + +Two passes, run during compile: + +- **Pass 1 — measure.** Walk in order; each node's output latency is the max of its inputs' + latencies plus its own. Pure bookkeeping, no delays inserted. +- **Pass 2a — align every multi-input node.** At any node that has more than one audio input + port, delay the *faster* inputs up to the slowest one, so they don't comb-filter at the + summing point. This is **always on and is essentially free in absolute terms** — it only makes + the other inputs share a delay the slow input already forced (e.g. a lookahead-gated kick on a + drum bus). It's bounded by the spread *within that node*, never anything global. +- **Pass 2b — explicit latency groups.** You can name a set of output points that should leave + the engine equally delayed *even if they never meet at a bus or shared node* — e.g. a dry + signal to one physical output and an FX send that goes off to outboard, kept aligned. This is + the only opt-in part. A group member is an `(node, output port)`. + +Alignment-at-a-summing-point is automatic and universal; +alignment-across-unrelated-paths is explicit and rare. Per-bus alignment is bounded by +intra-bus spread. Low absolute latency comes from topology — a monitor mix branching off +upstream of a heavy Broadcast limiter just never inherits +that latency because latency only propagates forward along edges. And because delay is a +per-*edge* property, a single mic feeding both a fast monitor bus and a high-latency Broadcast bus +keeps its monitor edge fast and only its FOH edge delayed. + +**Delay lines** are ordinary nodes (`DelayNode`, inserted by the PDC pass) whose internal +state is a ring buffer of past samples. This ring is **node-internal state** in the sense +of [§5.1](#51-nodes-ports-lanes): it belongs to the node, it persists across blocks, and the compiler does not +touch it. It is distinct from the per-edge buffers the compiler hands out — those are +single-block carriers between nodes, while the delay line's ring spans many blocks. The +ring is sized to a fixed maximum (`MAX_PDC_SAMPLES`, tunable) so it never reallocates, +and its contents survive recompiles like any other node state. When the needed delay +*changes* on a recompile, the node **ramps** to the new length rather than stepping, so +there is no click. (One honest consequence: alignment is exact in steady state and +converges over a few blocks after an edit. Fine for human-speed changes.) + +If a path needs more compensation than the ring can hold, the compile is **rejected and a +specific error is surfaced to the operator** ("PDC over budget on path X: needs N ms, +max M ms"). The trigger is pathological (>1s of compensation), but it must never be a +silent ignored edit — at a show, "I patched it and nothing happened, no idea why" is the +worst outcome. + +--- + +## 9. Multi-core + +The DSP runs on one pinned core today and the flat execution list assumes that. The +proposal keeps the `Node` interface free of cross-node side effects, so a future scheduler +could run independent branches on the other pinned cores without rewriting a single node. +This is not built now, but the door is left open. + +--- + +## 10. What this proposal deletes or changes + +- `AudioPipe` (linked list, per-node mutex + queue) → `Node` + compiled graph. +- `AudioEngine`'s 64-slot scan under a shared mutex → RCU pointer swap of a compiled graph. +- `AudioInMtx` → ingress matrix + jitter rings. +- `SampleStream` (`std::queue`) → lock-free SPSC jitter ring. +- `AudioInPipe` route-filter → ingress routing table. +- `AudioSendMtx` / `AudioDirectOut` → egress matrix + send ring. +- `ControlPipeCreate` (linear: channel + stack position) → node-create + edge-create + + graph-revision framing, sent as declarative full state. +- `AudioData` → adds a bundled multi-channel frame (dissector update too). +- Plugin ABI: `AudioPipe` → `Node` (clean break; `core_eq`/`core_comp` and the UI desc side + migrate together, and the desc side learns about ports). + +--- + +## 11. Open questions + +Items flagged in the document that are not yet decided: + +1. **`MAX_PDC_SAMPLES` value.** + + The per-delay-node ring size that bounds how much latency compensation can be inserted + on any one edge before the compile is rejected. Needs a concrete default. + Drives the worst-case memory per delay line and therefore + the worst-case memory of a graph that uses many delay nodes. + +2. **Async sample-rate converter for clock drift.** + + The proposal handles small clock-drift accumulation in the jitter ring by skipping or + inserting one sample at a watermark crossing ([§6.2](#62-network-arrival-alignment-not-latency-compensation)). This is cheap and adequate when + clock sync is doing its job, but is audibly imperfect under sustained drift. A proper + asynchronous sample-rate converter is a much later component with its own design; + flagged here only so the proposal does not get read as "this is the long-term answer". + +3. **Wire layout for the evolved control packets.** + + The new node-create / edge-create / graph-revision packets are described in prose in + [§7](#7-compiling-and-reconfiguring-the-graph) but their specific byte layout is not pinned down. The encoding is the OSCP opaque + payload format (see [cluster-coordination.md](cluster-coordination.md) and [state-coordination-protocol.md](state-coordination-protocol.md)); concrete + schema is a follow-up. The Wireshark dissectors will need updating to match whatever + encoding is chosen. diff --git a/proposals/state-coordination-protocol.md b/proposals/state-coordination-protocol.md new file mode 100644 index 0000000..52a5663 --- /dev/null +++ b/proposals/state-coordination-protocol.md @@ -0,0 +1,979 @@ +# OAN State Coordination Protocol (OSCP) + +**Status:** Proposal, not implemented, not yet reviewed +**Author:** Jonathan Reichardt +**Version:** Draft 0.2 +**Last updated:** 2026-06-01 +**Audience:** OALS contributors (current and future), and anyone implementing or porting OAN. +**Reading order:** third of three. Read [engine-audio-pipeline.md](engine-audio-pipeline.md) and [cluster-coordination.md](cluster-coordination.md) first; this proposal is best understood as the wire-level mechanism the cluster proposal calls for. See [architecture-overview.md](architecture-overview.md) for the rationale. +**Prerequisites:** familiarity with OAN at the level of `OpenAudioNetwork/common/packet_structs.h` and `OpenAudioNetwork/netutils/LowLatSocket.h`. The cluster proposal motivates the requirements OSCP is built to satisfy. + +## Abstract + +This document proposes OSCP, a distributed configuration synchronisation +protocol that would run alongside the existing OpenAudioNetwork (OAN) audio, +control, mapping, and clock-sync streams on a single L2 segment. The +proposal covers a single, consistent, system-wide configuration state across +all OAN peers; atomic transactional updates with sample-accurate apply +timestamps; single-authority election and authority failover; late-joiner +state synchronisation; and partition tolerance with post-heal +reconciliation. OSCP is deliberately **not** an OAN control plane: +per-element parameter updates (faders, EQ, dynamics, routing matrix entries) +continue to use the existing `ControlPacket` mechanism and never transit +OSCP. The protocol is opaque to the payload it carries — [cluster-coordination.md](cluster-coordination.md) +defines one thing OALS specifically puts inside it. + +## 1. Scope and non-goals + +### 1.1 In scope + +- System-wide configuration that must be **consistent across all peers**: + for example sample rate, clock-sync profile selection, network-wide + enables, show identity, network-level policy flags. The OALS-specific + payload (cluster blueprint) is one realisation; OSCP itself does not + define it. +- Atomic, transactional commit of such configuration with sample-accurate + apply timestamps. +- Single-authority election and authority failover. +- Late-joiner state synchronisation. +- Partition tolerance and post-heal reconciliation. +- In-protocol fragmentation of payloads larger than one L2 MTU ([§6.4](#64-in-protocol-fragmentation)). + +### 1.2 Out of scope + +- Per-element parameter updates (faders, gains, EQ, dynamics, routing + matrix entries, node parameters): handled by existing `ControlPacket` + traffic on `ETH_PROTO_OANCONTROL`. These are latency-critical, + high-frequency, lossy-tolerant, and explicitly **not** subject to OSCP's + transactional machinery. +- Audio sample transport: handled by `AudioPacket` on `ETH_PROTO_OANAUDIO`. +- Clock synchronisation: handled by `ClockMaster`/`ClockSlave` on + `ETH_PROTO_OANSYNC`. +- Peer discovery: handled by `NetworkMapper` on `ETH_PROTO_OANDISCO`. +- Persistence of any state, anywhere. OSCP state is **session-only**; see + [§5.4](#54-lifecycle-states-and-session-only-persistence). Persistent application data (show files, presets) is the + responsibility of separate application-layer mechanisms. Such + mechanisms may use `OSCP_PROPOSE` as one transport for re-injecting + persisted data into a freshly-booted network, but the act of saving and + loading is not part of this protocol. +- The schema, encoding, or semantic content of the configuration payload. + OSCP treats configuration as an opaque byte blob with a length prefix. + The application layer (OALS engine, coreui, io_sim, etc.) defines and + interprets its contents. [cluster-coordination.md](cluster-coordination.md) describes the + OALS-specific schema. + +## 2. Glossary + +Terms specific to this protocol. Cross-document terms (OAN, EtherType, peer, +UID, engine, UI, IO board, block, `apply_at_us`) are defined once in the +shared glossary in [architecture-overview.md §3](architecture-overview.md#3-shared-glossary). + +- **Authority** — the peer currently elected as OSCP coordinator. Receives + proposals, validates them, orchestrates PREPARE/COMMIT, broadcasts state. + Exactly one authority per partition at any time. +- **Proposer** — any peer (typically a control surface or operator UI) + submitting a proposed configuration change via PROPOSE. Any active peer + may propose. +- **Config state** — the opaque payload representing the current system + configuration. Identified by `(generation, version)`. +- **Generation** — monotonic counter incremented on every authority change. +- **Version** — monotonic counter incremented on every committed change + within a generation. +- **Transaction id** — per-transaction correlation identifier chosen by the + proposer. Also reused as the fragmentation-reassembly key ([§6.4](#64-in-protocol-fragmentation)). +- **Apply timestamp** (`apply_at_us`) — a future clock-disciplined time at + which all peers simultaneously enact a committed change. Defined in + [architecture-overview.md §3](architecture-overview.md#3-shared-glossary); the mechanism is described in [§8.4](#84-apply-scheduling-and-commit-finality) below. +- **Partition** — a subset of peers mutually reachable on the L2 segment. +- **Lifecycle state** — per-peer OSCP runtime state; see [§5.4](#54-lifecycle-states-and-session-only-persistence). + +## 3. Transport + +OSCP runs as a new EtherType on top of the existing `LowLatSocket` raw +Ethernet transport. + +| EtherType | Name | Purpose | +|-----------|------|---------| +| `0x0685` | `ETH_PROTO_OANSTATE` | OSCP messages | + +All OSCP messages are encapsulated as `LowLatPacket`, sharing +the existing `LowLatHeader { sender_uid, dest_uid, psize }` framing. +Broadcast messages use `dest_uid = 0`; targeted messages use the +recipient's UID. + +OSCP messages share the `CommonHeader` structure used by other OAN packets, +with `type` taking new `PacketType` values defined in [§6](#6-message-types). + +## 4. Architectural overview + +The proposal is **single-leader, broadcast-with-correlation**: + +- One **authority** per partition, elected by a BMCA-style algorithm with + its own data set distinct from the clock BMCA. +- All configuration changes flow **through** the authority. Other peers do + not directly mutate shared state. +- Any peer may **propose** a change. The authority validates, runs a + PREPARE/COMMIT round against all peers, and broadcasts the outcome. +- Outcomes (COMMIT or ABORT) are **broadcast**, not unicast. Proposers + correlate outcomes to their requests via `transaction_id`. + +The authority's role is restricted to: (a) being a single serialisation +point for state changes, and (b) rebroadcasting the agreed-upon state. It +is not a unilateral decision-maker; any peer can NACK during PREPARE. + +OSCP composes with the existing OAN infrastructure rather than replacing +any of it. It assumes peer discovery, clock sync, and raw-L2 transport are +already in place. + +> **Alternatives considered — multi-leader / CRDT.** A multi-leader design +> would let any peer accept proposals and reconcile divergence using +> CRDTs or operational transforms, removing the single-authority +> bottleneck and the authority-failover transient. This proposal rejects +> it for two reasons. (1) The OALS use case wants explicit serialisation +> at commit time so that an `apply_at_us` is meaningful — there must be +> one agreed-upon committed state per instant, not a merge-after-the-fact +> view. (2) The kinds of state OSCP carries (cluster blueprint, sample +> rate, clock profile) are not naturally CRDT-shaped; structural graph +> edits are not commutative. The cost of the single-leader choice is the +> authority-failover transient (seconds at worst, during which structural +> edits are paused). The cluster proposal absorbs this cost by keeping +> audio and parameter edits independent of the authority while it is +> being re-elected — see [cluster-coordination.md §8.3](cluster-coordination.md#83-authority-dies-audio-keeps-flowing). + +## 5. State model + +### 5.1 Per-peer state + +Every peer maintains: + +``` +struct OscpLocalState { + Generation generation; // uint64, current + Version version; // uint64, current + uint16_t authority_uid; // who set the current state + uint64_t last_change_us; // PTP time of last accepted commit + Blob payload; // opaque config bytes +}; +``` + +### 5.2 State identity and ordering + +Two states are compared by lexicographic `(generation, version)`. A state +with higher `generation` always wins regardless of `version`. Within a +generation, higher `version` wins. + +`generation` increases when the authority changes. `version` increases on +each committed change. + +### 5.3 Show-state vs device-state + +OSCP synchronises only **show-state**: configuration that must be +consistent across the network. **Device-state** (hardware capabilities, +device UID, factory calibration, persistent identity) is local to each +peer, not transported by OSCP, and never overwritten by OSCP transactions. +Device-state is already covered by OAN mapping packets (`MappingData`, +`NodeTopology`). + +The split is enforced by convention: the application layer decides what +goes into the OSCP payload and what stays local. OSCP itself does not +distinguish. + +### 5.4 Lifecycle states and session-only persistence + +This proposal makes OSCP state **session-only**. No OSCP-managed state +persists across peer reboots. On boot, every peer starts with no payload, +no generation, no version, and no priority assignment beyond the +device-type default ([§7.2](#72-default-priority-ladder)). + +The intent is to eliminate the "stale-config-from-last-gig" failure mode +entirely: a freshly booted device cannot inject configuration from a +previous network into the current one, because it has none to inject. + +Re-establishing configuration on a freshly powered network is an +application-layer concern handled by mechanisms outside OSCP (show files, +preset databases). Such mechanisms may use `OSCP_PROPOSE` as one delivery +vehicle for re-injecting saved configuration into the running network, +but the act of saving, loading, and choosing what to restore is not part +of this protocol. + +> **Alternatives considered — persistent OSCP state.** A persistent +> design would have peers store the last-committed `(generation, version, +> payload)` to disk and reload it on boot. The appeal is faster recovery +> after a full power-cycle: the network can come up at the same state it +> went down in, without needing a UI to re-propose. The proposal rejects +> this for the failure mode it introduces. A device powered up at a new +> venue still holding state from the previous gig is a real and +> consequential bug — it can win election against the actual current +> show, project a stale blueprint, and the operator has no obvious cue +> that anything is wrong. Session-only state is a hard guarantee against +> that class of failure. The cost paid (a fresh network has no audio +> until a UI joins and proposes a blueprint) is acceptable for the +> case OALS primarily targets; fixed-install scenarios can be +> served by a small auto-loader peer that proposes a saved blueprint on +> boot. The auto-loader peer would be an application-layer addition, not +> a change to OSCP. + +Each peer transitions through the following lifecycle: + +``` + boot + │ + ▼ + ┌─────────────┐ + │ UNCONFIGURED│ no payload, no (gen, ver), no authority known + └──────┬──────┘ + │ start OSCP, listen for OSCP_ANNOUNCE + ▼ + ┌─────────────┐ + │ LISTENING │ waiting up to T_listen for an announce + └──────┬──────┘ + │ + ├── announce received from existing authority + │ ──▶ ┌─────────────┐ + │ │ JOINING │ STATE_REQUEST → adopt payload + │ └──────┬──────┘ + │ │ STATE received & applied + │ ▼ + │ ┌─────────────┐ + │ │ ACTIVE │ participate normally, eligible + │ └─────────────┘ to propose, ACK/NACK PREPAREs, + │ and stand for election + │ + └── T_listen expires with no announce + ──▶ ┌─────────────┐ + │ ACTIVE │ become authority with EMPTY state + │ (authority)│ (generation = 1, version = 0, + └─────────────┘ payload = empty). Accept first + PROPOSE as initial config. +``` + +Only ACTIVE peers participate in OSCP transactions and elections. A peer +in `UNCONFIGURED`, `LISTENING`, or `JOINING` does not propose changes and +does not stand for election. + +Figure 1: peer lifecycle from boot to ACTIVE participation. + +## 6. Message types + +| `PacketType` (proposed) | Direction | Reliability | Purpose | +|------------------------|-----------|-------------|---------| +| `OSCP_ANNOUNCE` | broadcast | periodic (1 Hz) | Authority advertises election data set and current `(generation, version)`. | +| `OSCP_PROPOSE` | unicast → authority | retransmitted by proposer if needed | Request a state change. | +| `OSCP_PROPOSE_ACK` | unicast → proposer | one-shot | "I received your PROPOSE." Optional UX hint. | +| `OSCP_PREPARE` | broadcast | retransmitted by authority | Authority distributes a candidate state and tx_id. | +| `OSCP_PREPARE_ACK` | unicast → authority | one-shot | Per-peer ACK/NACK with reason. | +| `OSCP_COMMIT` | broadcast | retransmitted by authority | All peers apply the candidate state at `apply_at_us`. **Final** — see [§8.4](#84-apply-scheduling-and-commit-finality). | +| `OSCP_ABORT` | broadcast | retransmitted by authority | All peers discard candidate state. Carries the first NACK or timeout that triggered the abort. | +| `OSCP_STATE_REQUEST` | unicast → authority | one-shot | Late joiner or out-of-date peer requests full current state. | +| `OSCP_STATE` | unicast → requester | response | Authority replies with full payload, optionally including `heal_info` ([§9.4](#94-heal-info-on-state-messages-ux-recommendation)). | + +Periodic `OSCP_ANNOUNCE` transmissions provide loss recovery: a peer that +missed a commit but receives an announce with a higher +`(generation, version)` than its own issues `OSCP_STATE_REQUEST`. + +Any of these messages whose serialised form exceeds the on-wire frame +budget is split into fragments and reassembled per [§6.4](#64-in-protocol-fragmentation). The fragment +mechanism is shared across all message types, not specific to PREPARE. + +### 6.1 Common transaction header + +All transactional messages (`PROPOSE`, `PREPARE`, `PREPARE_ACK`, `COMMIT`, +`ABORT`) share a common header: + +``` +struct OscpTxHeader { + uint64_t transaction_id; // proposer-chosen, unique + uint16_t proposer_uid; // origin of the proposal + uint16_t authority_uid; // authority handling it (0 in PROPOSE) + uint64_t base_generation; // (generation, version) the proposer based on + uint64_t base_version; +} __attribute__((packed)); +``` + +### 6.2 NACK / ABORT reason codes + +``` +enum OscpReason : uint16_t { + OSCP_OK = 0, + OSCP_NACK_STALE_BASE = 1, // base_(gen,ver) doesn't match current + OSCP_NACK_BUSY = 2, // authority has in-flight tx + OSCP_NACK_TIMEOUT = 3, // peer failed to ACK PREPARE in time + OSCP_NACK_UNSUPPORTED = 4, // peer cannot honor payload (capability) + OSCP_NACK_INVALID_PAYLOAD= 5, // schema / parse failure + OSCP_NACK_BAD_APPLY_TIME = 6, // apply_at_us out of acceptable window + OSCP_NACK_AUTHORITY_LOST = 7, // authority changed mid-transaction + OSCP_NACK_INTERNAL = 255 // peer-specific failure +}; +``` + +Application-layer rejection reasons extend this enum in a range reserved +for application use (`0x8000`–`0xFFFF`). The protocol-defined range stops +at `0xFF`; the gap is reserved for future protocol additions. + +### 6.3 Message payloads + +Definitions are illustrative; final field widths and ordering are TBD in +Section 11. + +``` +struct OscpAnnounce { + OscpElectionData election_data; // see §7.1 + uint64_t generation; + uint64_t version; + uint64_t last_change_us; + uint32_t peer_count; // peers seen by this authority +} __attribute__((packed)); + +struct OscpPropose { + OscpTxHeader tx; + uint32_t payload_len; + uint8_t payload[]; // proposed new state +}; + +struct OscpProposeAck { + OscpTxHeader tx; +}; + +struct OscpPrepare { + OscpTxHeader tx; + uint64_t apply_at_us; // proposed apply timestamp + uint32_t payload_len; + uint8_t payload[]; +}; + +struct OscpPrepareAck { + OscpTxHeader tx; + uint16_t peer_uid; + uint16_t reason_code; // OscpReason + char reason_msg[64]; // human-readable, optional +} __attribute__((packed)); + +struct OscpCommit { + OscpTxHeader tx; + uint64_t apply_at_us; // same as in PREPARE + uint64_t new_generation; + uint64_t new_version; +} __attribute__((packed)); + +struct OscpAbort { + OscpTxHeader tx; + uint16_t peer_uid; // peer whose NACK / timeout triggered the abort + uint16_t reason_code; // OscpReason +} __attribute__((packed)); + +struct OscpStateRequest { + uint64_t current_generation; // what the requester knows + uint64_t current_version; +}; + +struct OscpState { + uint64_t generation; + uint64_t version; + uint16_t authority_uid; + uint64_t last_change_us; + uint8_t has_heal_info; // 0 or 1; see §9.4 + uint32_t payload_len; + uint8_t payload[]; + // if has_heal_info == 1, immediately followed by: + // uint32_t discarded_payload_len; + // uint8_t discarded_payload[]; + // uint64_t discarded_generation; + // uint64_t discarded_version; +}; +``` + +Variable-length payloads (`payload[]`) are length-prefixed. OSCP imposes +no logical maximum on opaque payload size; messages whose serialised +length exceeds one L2 frame are fragmented per [§6.4](#64-in-protocol-fragmentation). + +### 6.4 In-protocol fragmentation + +A realistic OALS cluster blueprint (see `cluster-coordination.md`) +is comfortably larger than one Ethernet frame — dozens of nodes, edges, +and per-node metadata land in the 20–50 KB range, which is 15–30× the +standard 1500-byte MTU. The proposal pushes fragmentation into OSCP +itself rather than into the application layer above it. The reason is +atomicity: an OALS blueprint commit must be all-or-nothing. If +fragmentation were the application's responsibility, an "apply the +blueprint" operation would either span multiple OSCP transactions +(losing the atomicity guarantee that is the central reason for using +OSCP at all) or be constrained to fit in one frame (which it does not). +Putting fragmentation inside OSCP keeps the proposal honest: one +PROPOSE / PREPARE / COMMIT, one apply instant, regardless of payload +size. + +**Fragment header.** Each fragment of a transactional message carries, +in addition to the normal `OscpTxHeader`, a fragment header: + +``` +struct OscpFragmentHeader { + uint32_t fragment_seq; // 0..fragment_total-1, in transmission order + uint32_t fragment_total; // total number of fragments in this message + uint32_t payload_offset; // byte offset of this fragment's bytes within the full payload + uint32_t this_fragment_len; // length of this fragment's payload bytes +} __attribute__((packed)); +``` + +Single-frame messages omit the fragment header entirely (the message +type byte signals "no fragmentation"); only payloads that overflow one +frame carry it. + +**Reassembly.** Each peer maintains a reassembly buffer keyed by +`(authority_uid, transaction_id)`. Fragments belonging to one +transaction are written into the buffer at `payload_offset`. A +reassembly entry is considered complete when every byte in +`[0, payload_len_total)` has been received and the count of distinct +fragments matches `fragment_total`. Stale partials time out after a few +seconds (recommended `T_reassembly = 5 s`) and are discarded. + +**ACK timing.** A peer ACKs PREPARE only when reassembly is complete *and* +local validation passes. An incomplete reassembly produces a missing ACK +at the authority, which retransmits the full PREPARE; receivers ignore +duplicate fragments already in their buffer. This reuses the +retransmit-on-no-ACK loss-recovery path; no NACK-based +selective-retransmit machinery is needed. The cost is one full-message +retransmission on any fragment loss; acceptable, since structural-edit +volume is human-paced (seconds between transactions) and fragmentation +only kicks in for the rare cluster-wide reconfiguration. + +**Frame sizing — jumbo frames are not assumed.** Fragments are sized to +fit in **the standard 1500-byte Ethernet MTU minus headers** (Ethernet, +`LowLatHeader`, `CommonHeader`, `OscpTxHeader`, `OscpFragmentHeader`, +and the message-type-specific header). The protocol does not depend on +jumbo frames anywhere. The reasons are deployment-environment +predictability: + +- Every NIC, switch, and bridge on the L2 path must agree on the chosen + MTU. Oversized frames are dropped — often silently — by equipment + that does not expect them. Common deployment environments break the + assumption: consumer NICs defaulting to 1500, Wi-Fi access points + (the wireless side rarely supports jumbos), unmanaged switches, + USB-Ethernet adapters. +- Even where hardware supports jumbos, larger frames trade head-of-line + blocking against overhead savings. A 9000-byte frame takes ~72 µs to + serialise at gigabit Ethernet vs. ~12 µs for a 1500-byte frame. The + same OAN segment carries audio packets at the per-block budget of + ~667 µs, and head-of-line blocking from a jumbo OSCP fragment would + consume a non-trivial slice of that budget. + +A future extension can add per-peer MTU advertisement in `MappingData`, +let the authority size fragments to the cluster-minimum MTU, and grow +the OSCP frame on segments where every peer supports jumbos. The +present proposal stays at 1500 bytes to keep behaviour identical across +deployments. See [cluster-coordination.md §3.4](cluster-coordination.md#34-mtu-and-jumbo-frames) for the +corresponding choice on the audio path. + +**Per-peer reassembly-buffer cap.** A bounded cap on the total memory a +peer holds in unresolved reassembly buffers is needed to prevent a buggy +or hostile authority from exhausting peer memory with very large +`fragment_total` advertisements. The exact cap is left as an open +question ([§11](#11-open-questions)); a few-MiB cap per `(authority_uid, transaction_id)` is +likely sufficient given the realistic blueprint sizes named above. + +## 7. Authority election + +### 7.1 Election data set + +Each peer carries an OSCP-specific election data set: + +``` +struct OscpElectionData { + uint8_t config_priority_1; // operator-set, lower = wins (see §11) + uint8_t config_priority_2; // operator-set tiebreaker (see §11) + uint16_t device_role; // enum, see §7.2 + uint32_t peer_count; // peers visible to this candidate + uint64_t last_change_us; // most recent operator-initiated change + uint64_t device_identity; // typically derived from MAC +} __attribute__((packed)); +``` + +User-settable priorities are **session-only** ([§5.4](#54-lifecycle-states-and-session-only-persistence)): they reset to the +device-type default on every boot. The intent is to eliminate the most +dangerous class of stale-priority bug — a high-priority device from a +previous show auto-winning election on a new network without operator +awareness — while preserving the operator's ability to elevate a +specific peer within a running session. Whether `config_priority_1` +should remain user-settable at all, or be reduced to a synchronised +session-state value managed via OSCP itself, is unresolved ([§11](#11-open-questions)). + +### 7.2 Default priority ladder + +`config_priority_1` defaults by role, lowest = most authoritative. The +table below is the OALS-specific ladder this proposal recommends; the +cluster proposal ([cluster-coordination.md §3.2](cluster-coordination.md#32-oals-overrides-and-dependencies-on-oscp)) explains the +reasoning behind placing engines above UIs. + +| Role | Default `config_priority_1` | +|------|----------------------------| +| Engine / DSP node | 32 | +| Operator UI / control surface | 100 | +| io card / stage box | 200 | +| Monitor-only / VSC bridge | 240 | +| Slave-only (never authority) | 255 | + +Operators can override these via local configuration **for the current +session only** — the override does not persist across reboots. + +### 7.3 Comparison algorithm + +Two `OscpElectionData` values are compared lexicographically by: + +1. `config_priority_1` (lower wins) +2. `peer_count` (higher wins) +3. `last_change_us` (higher / more recent wins) +4. `config_priority_2` (lower wins) +5. `device_identity` (lower wins, final tiebreaker) + +The lower-numbered field always dominates; subsequent fields are +tiebreakers only. + +### 7.4 Election state machine + +Election runs only while a peer is `ACTIVE` ([§5.4](#54-lifecycle-states-and-session-only-persistence)). The states below are +substates of `ACTIVE`: + +``` +ASPIRANT (transient) — competing for authority + └─ on receipt of better-quality announce → SLAVE + └─ on becoming uncontested for T_listen → MASTER + +MASTER broadcast OSCP_ANNOUNCE every T_announce + └─ on receipt of better-quality announce → SLAVE + └─ on operator priority change that worsens position → may step down + +SLAVE listening + └─ on T_announce_timeout × N without announce from current authority + → re-enter ASPIRANT + └─ on receipt of a better-quality announce → re-target authority +``` + +Recommended timers: + +- `T_announce` = 1 s +- `T_listen` = 3 × `T_announce` +- `T_announce_timeout` = 3 × `T_announce` + +### 7.5 Authority change + +When the authority changes: + +1. The new authority bumps `generation := generation + 1`. +2. Before serving any state, the new authority issues `OSCP_STATE_REQUEST` + to a small set of peers (or all) to discover the most recent known + state. +3. The new authority adopts the highest-`(generation, version)` state + observed, bumps `generation` to one greater than any observed, and + broadcasts the resulting state via `OSCP_ANNOUNCE`. +4. Pending transactions targeting the previous authority are implicitly + aborted; proposers detect this via `OSCP_ANNOUNCE` carrying a new + `authority_uid` and resubmit if the proposal is still relevant. + +## 8. Transaction flow + +The canonical successful flow: + +``` +proposer authority all peers + │ │ │ + │ PROPOSE(tx) │ │ + │ ──────────────────▶│ │ + │ │ PROPOSE_ACK(tx) │ (optional, UX hint) + │ ◀──────────────────│ │ + │ │ │ + │ │ PREPARE(tx, new, │ + │ │ apply_at_us) │ + │ │ ─────broadcast──▶ │ + │ │ │ + │ │ PREPARE_ACK(tx,…) │ + │ │ ◀──────────────── │ (one per peer) + │ │ │ + │ │ (collect ACKs; │ + │ │ fail-fast on │ + │ │ any NACK) │ + │ │ │ + │ │ COMMIT(tx, │ + │ │ apply_at_us) │ + │ │ ─────broadcast──▶ │ + │ │ │ + │ │ │ All peers schedule + │ │ │ apply at apply_at_us. + │ │ │ At that instant, + │ │ │ payload is enacted. +``` + +Figure 2: canonical OSCP transaction flow. + +### 8.1 Proposer responsibilities + +- Generates a unique `transaction_id` per proposal. Recommended: + `(proposer_uid << 48) | (startup_epoch & 0xFFFF) << 32 | counter`. +- Stamps `base_generation` and `base_version` to the last observed state. +- Retransmits PROPOSE if no PROPOSE_ACK or COMMIT/ABORT is observed within + `T_propose_retry` (recommended 200 ms). Limit retries (recommended 5). +- Tracks pending transactions in a local map indexed by `transaction_id`. +- On receipt of COMMIT or ABORT matching a pending `tx_id`, resolves the + pending entry. +- On `T_propose_timeout` without resolution (recommended 5 s), resolves + locally with `Status::TIMEOUT`. +- On `OSCP_NACK_BUSY` or `OSCP_NACK_STALE_BASE`: the proposer's baseline + is stale. The proposer waits for the next COMMIT or ANNOUNCE, + re-derives its proposal against the updated baseline (an + application-layer concern — the new baseline may invalidate the + proposal entirely, may need merging, or may need to be presented back + to the operator), and only then resubmits. Blind retry against the + old baseline will continue to fail. + +### 8.2 Authority responsibilities + +- Accepts at most one in-flight transaction at a time. New proposals + while a transaction is in flight receive immediate `OSCP_ABORT` with + reason `OSCP_NACK_BUSY`. +- New proposals carrying a `base_(generation, version)` that does not + match the authority's current state receive immediate `OSCP_ABORT` + with reason `OSCP_NACK_STALE_BASE`. +- On PROPOSE receipt, optionally emits PROPOSE_ACK immediately. +- Validates the proposal locally first (schema, invariants). On local + rejection, broadcasts `OSCP_ABORT` immediately with + `(peer_uid = authority_uid, reason)`. +- On local acceptance, broadcasts `OSCP_PREPARE` with + `apply_at_us := now + T_apply_offset` (recommended 50–100 ms). + `apply_at_us` is fixed at PREPARE time and carried unchanged through + COMMIT. +- Collects `OSCP_PREPARE_ACK` for up to `T_prepare_timeout` (recommended + 500 ms). The set of peers expected to respond is the authority's + current `NetworkMapper` view at the moment PREPARE was broadcast. + Each peer's `OSCP_PREPARE_ACK` includes its own `peer_uid` so the + authority can identify who has responded; no explicit peer set is + carried in PREPARE. +- The **quorum policy** — which set of peers must ACK before the + authority can COMMIT — is authority-configurable rather than fixed by + the protocol. Two policies the proposal expects to be common: + - **Strict** (default suggested): every peer in the authority's view + must ACK; any missing response after `T_prepare_timeout` is treated + as NACK with reason `OSCP_NACK_TIMEOUT`. Safest "everyone agrees" + semantics. + - **Lenient**: proceed if a configurable quorum (for example + authority + majority) ACKed; missing peers will resynchronise via + `OSCP_STATE_REQUEST` after observing the next ANNOUNCE. + + The choice is itself a configuration value and can therefore be set + via OSCP after initial bootstrap. The alternatives callout below + explains why the protocol does not mandate one. +- **Fails fast on the first NACK.** As soon as one peer responds with + `reason_code != OSCP_OK`, the authority broadcasts `OSCP_ABORT` + carrying that peer's `(peer_uid, reason_code)`, abandons aggregation, + and is ready for the next proposal. No aggregated NACK list is + carried; if other peers were also going to reject for the same + reason, they will reject the next proposal too. +- If `T_prepare_timeout` elapses with at least one peer still not + having ACKed under strict quorum, broadcasts `OSCP_ABORT` with + `reason_code = OSCP_NACK_TIMEOUT` and `peer_uid` set to one of the + silent peers. Under lenient quorum, proceeds with COMMIT once the + configured quorum has ACKed. +- If all required ACKs arrive with `reason_code == OSCP_OK` before + `T_prepare_timeout`, broadcasts `OSCP_COMMIT`. +- Bumps `version` only on successful COMMIT. + +> **Alternatives considered — protocol-mandated quorum.** A simpler +> design would pick one quorum policy and bake it into the protocol: +> either strict-everyone-ACKs (safe but brittle when one peer goes +> silent) or fixed-majority (robust to one slow peer but +> "majority-of-what" is genuinely contested in a single-segment system +> with heterogeneous peer counts). This proposal leaves the choice as +> authority configuration because the right answer is +> deployment-dependent and a protocol-wide mandate would be wrong for +> one case or the other. The cost is one more configuration knob; +> mitigated by giving the strict policy as the default so the +> conservative case requires no configuration. + +### 8.3 Peer responsibilities + +- On `OSCP_PREPARE`, validate locally (hardware capability, schema, + `apply_at_us` reasonableness). Reply unicast `OSCP_PREPARE_ACK` with + `reason_code = OSCP_OK` or an appropriate NACK code. +- On `OSCP_COMMIT` matching a PREPARE the peer ACKed, schedule + application for `apply_at_us`. At that timestamp, enact the prepared + payload and update local `generation`/`version`. +- On `OSCP_ABORT`, discard the candidate payload. +- On `OSCP_ANNOUNCE` indicating a `(generation, version)` newer than + local, issue `OSCP_STATE_REQUEST` to the authority. + +### 8.4 Apply scheduling and COMMIT finality + +This section names normative behaviour of the protocol. The MUST / MUST +NOT language is intentional and is the reason atomic-apply works at all. + +`apply_at_us` is expressed in OAN clock-disciplined time. Peers schedule a +local timer (via `clock_nanosleep` against `CLOCK_MONOTONIC` adjusted by +the current clock-slave offset, or against a future PTP/gPTP-disciplined +clock once available) to enact the change at the specified instant. + +Peers MUST reject `apply_at_us` values that are in the past or +unreasonably far in the future (recommended bound: 2 s). On rejection, +emit a NACK during PREPARE (reason `OSCP_NACK_BAD_APPLY_TIME`). By the +time COMMIT is on the wire, it is too late to reject. + +**COMMIT is final.** Once an `OSCP_COMMIT` has been broadcast for a +given `transaction_id`, no rollback is possible. The window between +COMMIT and `apply_at_us` is purely a scheduling mechanism, not a +decision mechanism. Specifically: + +- A NACK arriving after COMMIT (e.g., delayed by network reorder) MUST + be ignored by the authority and SHOULD be logged as a misbehaving + peer. +- A peer that ACKed PREPARE but is unable to enact the payload at + `apply_at_us` (rare hardware fault) MUST enter a local degraded state + and resynchronize via `OSCP_STATE_REQUEST` on the next ANNOUNCE. It + MUST NOT attempt to roll back the global commit. +- A peer that did not see PREPARE (joined mid-transaction, missed + packet) but receives COMMIT MUST NOT apply blindly; it issues + `OSCP_STATE_REQUEST` and adopts the resulting STATE. + +This guarantee — "after COMMIT is on the wire, the new state is the new +state" — is what allows applications to treat OSCP transactions as +atomic. + +## 9. Partition handling + +### 9.1 During partition + +Each partition independently re-runs election as needed (existing +authority unreachable triggers timeout). Each partition continues to +accept proposals and commit changes within its own scope. There is no +inter-partition coordination; partitions are unaware of each other by +design. + +A partition whose authority dies and that contains no other +authority-eligible peer (all remaining peers have +`config_priority_1 = 255`) refuses new proposals and surfaces a fault. +The default priority ladder ([§7.2](#72-default-priority-ladder)) ensures this case does not occur in +any normal deployment. + +A partition whose authority dies but that contains authority-eligible +peers re-elects normally. Election proceeds on remaining peers per [§7.4](#74-election-state-machine); +configuration changes resume once a new authority is established. State +carried across re-election is the previous authority's last broadcast +state, as preserved by the surviving peers. + +### 9.2 On heal + +When partitions reconnect (mapping packets re-establish mutual +visibility), both authorities emit `OSCP_ANNOUNCE`. The election +algorithm ([§7.3](#73-comparison-algorithm)) selects one. The losing authority transitions to SLAVE +and adopts the winning authority's state via `OSCP_STATE_REQUEST`. + +**Discarded state is not merged.** The losing partition's local +`(generation, version)` is overwritten unconditionally. OSCP itself does +not attempt to reconcile divergent changes; semantic merge is left to +application-layer mechanisms (which may, for example, present a diff and +offer a re-propose UI). + +### 9.3 Stale-config-on-rejoin (rendered moot by §5.4) + +The classic failure mode — a device arriving at a new venue with +leftover config from a previous gig and dominating the live network — is +eliminated by design via session-only state ([§5.4](#54-lifecycle-states-and-session-only-persistence)): a freshly booted +device has no payload to inject. It enters `LISTENING`, finds the +existing authority, joins, and adopts current state. + +The remaining failure mode is partition rejoin, where two halves of a +network that *both* contain peers with live state must reunite. This is +handled by [§9.2](#92-on-heal) and the heal-info UX described in [§9.4](#94-heal-info-on-state-messages-ux-recommendation). + +### 9.4 Heal-info on STATE messages (UX recommendation) + +To make discarded partition state visible to operators, the `OSCP_STATE` +message defined in [§6](#6-message-types) carries an optional `heal_info` field. When the +authority sends STATE to a peer that is adopting the winner's state +after a partition heal, it can include the losing partition's prior +`(generation, version)` and payload as `heal_info`. + +Recipient policy is an application-layer recommendation, not mandated by +OSCP: + +- **Headless peers (engines, io-cards)** log `heal_info` if present and + otherwise ignore it. +- **Operator-facing peers (control surfaces, UIs)** display a non-modal + notification: "Partition rejoined; N config changes made while + disconnected were discarded. [View diff] [Re-propose]". +- **Multiple operator UIs**: there is no concept of a "primary" UI. All + operator UIs receive the same `heal_info` and display the + notification independently. The first operator to act (dismiss, + re-propose) does so via normal OSCP traffic, which is observable by + the others. No special arbitration is required. +- **Networks with no operator UI present**: nobody surfaces the + notification. The heal silently completes. Engines and io-cards log + for post-mortem analysis. + +Presenting a "re-propose" UI and recomputing the discarded changes +against the new baseline is entirely the application's responsibility. +OSCP's contribution is to make the raw material — the discarded payload +— available to whoever wants it. + +### 9.5 Partition scenarios + +Illustrative scenarios under the rules above: + +**S1 — clean split, only one side mutates.** Surface + engine on side +A make 5 changes; stage box on side B (no proposer present) is +unchanged. On heal, A wins on `(generation, version)` recency. Nothing +is lost. + +**S2 — both sides mutate.** Surface + 2 io-cards on A; surface + 1 +io-card on B. Both make changes. On heal, A wins on `peer_count` +(3 > 2). B's changes are discarded; B's UI shows the heal notification +([§9.4](#94-heal-info-on-state-messages-ux-recommendation)). B's operator may re-propose. + +**S3 — wrong-network bystander.** Main rig (15 peers) on A; an +operator's test setup (2 peers) accidentally bridged in via B. On +heal, A wins on `peer_count`. B's test config is discarded — correct +outcome. + +**S4 — equal-size partitions.** Tiebreakers in order: `peer_count` +(tie), `last_change_us` (most recently active wins), then priority, +then `device_identity`. `last_change_us` correctly favors the side +that was actually in use. + +**S5 — authority-side partition shrinks.** Authority A's partition has +only A and one engine. The other partition (no surface) re-elects +locally (per [§9.1](#91-during-partition)) and continues. On heal, BMCA picks based on +`peer_count` and tiebreakers. The post-split side typically wins on +`peer_count`, which is the correct outcome — the side that *is the +show* should keep its state. + +**S6 — three-way split and pairwise reunion.** Reuniting partitions +pairwise reduces to repeated S1/S2/S4. No special three-way logic is +needed. + +**S7 — old authority rejoins, new authority elsewhere.** Surface A +(former authority) on a 3-peer partition; surface B elected new +authority on an 8-peer partition after losing contact with A. Both +sides have made changes. On heal, B wins on `peer_count`; A's +partition's changes are discarded. Operator on A sees the heal +notification. + +## 10. Failure modes and recovery + +| Scenario | Detection | Recovery | +|----------|-----------|----------| +| PROPOSE lost | No PROPOSE_ACK within `T_propose_retry` | Proposer retransmits up to N times. | +| PREPARE lost to one peer | Missing PREPARE_ACK from that peer at authority | Authority retransmits PREPARE to that peer; on continued silence after `T_prepare_timeout`, fail-fast ABORT with `OSCP_NACK_TIMEOUT` under strict quorum, or proceed under lenient quorum. | +| PREPARE_ACK lost | Authority's view shows missing ACK | Authority retransmits PREPARE (idempotent). | +| COMMIT lost to one peer | Peer never applies; subsequent ANNOUNCE shows newer state | Peer issues STATE_REQUEST. | +| Late NACK after COMMIT broadcast | Authority observes NACK with `tx_id` matching an already-committed transaction | Authority logs and ignores. COMMIT is final ([§8.4](#84-apply-scheduling-and-commit-finality)). | +| Peer fails to apply at `apply_at_us` (rare hardware fault) | Local error at apply time | Peer enters local degraded state and resynchronises via STATE_REQUEST. Does not attempt global rollback ([§8.4](#84-apply-scheduling-and-commit-finality)). | +| Authority crash mid-transaction | Heartbeat timeout in slaves | Re-election ([§7.4](#74-election-state-machine)); new authority issues STATE_REQUEST to discover surviving state; pending transactions implicitly aborted; proposers retry on the new authority. | +| Authority crash post-COMMIT but pre-broadcast | Lost commit; election occurs | New authority may not see the COMMIT; the transaction is effectively lost. Proposer times out and retries. See [§11](#11-open-questions). | +| Apply timestamp missed (clock skew) | Peer's local clock at COMMIT receipt is already past `apply_at_us` | Peer applies immediately, logs the skew as a fault. | +| Multiple concurrent proposers | Second PROPOSE arrives during in-flight tx, or carries stale baseline | Authority rejects with `OSCP_NACK_BUSY` or `OSCP_NACK_STALE_BASE`; second proposer rebases per [§8.1](#81-proposer-responsibilities) and retries. | +| Partition with no authority-eligible peers | All surviving peers have `config_priority_1 = 255` | Configuration changes refused; fault surfaced. Does not occur in default deployments ([§7.2](#72-default-priority-ladder), [§9.1](#91-during-partition)). | +| Partition with eligible peers but no authority (authority died) | T_announce_timeout × N elapses | Surviving peers re-elect ([§9.1](#91-during-partition)) and resume normal operation with last-broadcast state. | + +## 11. Open questions + +Items left deliberately unresolved in this draft and needing resolution +before implementation: + +1. **Fragmentation reassembly-buffer sizing.** [§6.4](#64-in-protocol-fragmentation) specifies in-protocol + fragmentation but leaves the per-peer reassembly buffer cap + unspecified. A bounded cap is needed to prevent a buggy or hostile + authority from exhausting peer memory with very large `fragment_total` + advertisements. A few-MiB cap per `(authority_uid, transaction_id)` + is likely right; the exact value should be pinned once the largest + realistic OALS blueprint payload (see + `cluster-coordination.md`) is sized concretely. + +2. **User-settable priority — keep or replace?** [§7.1](#71-election-data-set) keeps + `config_priority_1` as user-settable, session-only ([§5.4](#54-lifecycle-states-and-session-only-persistence) mitigates + the worst stale-priority failure mode). An alternative is to drop + user-settable priority entirely and use `device_role` class plus a + synchronised `assigned_priority` (managed via OSCP itself, also + session-only). The latter shifts operator UX from "set a number" to + "promote/demote" gestures and removes a sharp edge. Decision + deferred. + +3. **Authority crash between local COMMIT decision and COMMIT broadcast.** + The authority is the single serialisation point; a crash in this + window leaves the transaction effectively lost — the proposer times + out, no commit was observed. The proposal treats this as acceptable + for an initial implementation. A two-of-N commit log (Raft-style) + would close the gap, but the additional complexity is not justified + for OALS-scale clusters. + +4. **`peer_count` racing during partition heal.** Election uses + `peer_count` as a tiebreaker. During a heal in progress, both + authorities may observe transient peer counts before mapping + stabilises. Election convergence may oscillate. The proposal + suggests damping `peer_count` over a rolling window (for example + 5 s minimum residency before counting a peer); the actual mechanism + needs validation. + +5. **Anti-DoS on PROPOSE.** A malfunctioning or hostile peer can flood + the authority with PROPOSE messages. Per-source rate-limiting is the + obvious mitigation but is not specified here. + +7. **Authority handover atomicity for in-flight transactions.** [§7.5](#75-authority-change) + says pending transactions are implicitly aborted. An open question + is whether the new authority should explicitly broadcast + `OSCP_ABORT` for the orphaned transaction ids it inherits via + STATE_REQUEST. Doing so would give proposers fast feedback instead + of forcing them to wait for their local timeout. Likely yes, but + not specified. + +8. **Configuration namespacing.** The payload is opaque, but the + application layer will need conventions for evolving the schema + (adding fields, deprecating others) without coordinated upgrades. + Out of scope for OSCP itself; flagged here for the OALS application + layer. + +9. **PTPv2/gPTP wire-format adoption.** The clock side of OAN is + expected to evolve toward PTPv2-compatible message formats. OSCP's + `apply_at_us` semantics depend on a disciplined network clock; the + quality of that clock affects how tight `T_apply_offset` can be. + Tracked separately. + +10. **Heal-info delivery to non-authority operator UIs.** [§9.4](#94-heal-info-on-state-messages-ux-recommendation) places + `heal_info` on the STATE message sent to the (former) losing + authority. Operator UIs that were *not* the losing authority do + not receive `heal_info` directly. Options: (a) broadcast + `heal_info` after a heal, (b) every UI issues `STATE_REQUEST` + after a heal it observes, (c) the receiving (losing) authority + re-broadcasts a digest. Decision deferred; (b) is simplest if + STATE volume is acceptable. + +## 12. References + +- IEEE 1588-2008 — Precision Time Protocol v2 (BMCA algorithm; basis for + OSCP election logic). +- IEEE 802.1AS-2020 — gPTP (profile of PTPv2; relevant to OAN clock + evolution and `apply_at_us` accuracy). +- OAN protocol structures: `OpenAudioNetwork/common/packet_structs.h`. +- OAN transport: `OpenAudioNetwork/netutils/LowLatSocket.h`. + +## Appendix A — Implementation notes + +These notes are informative — they record implementation suggestions for +a future OSCP code drop in OAN, not normative requirements of the +protocol. + +- Reuse `LowLatSocket` for transport; add + `EthProtocol::ETH_PROTO_OANSTATE`. +- Reuse `OANPacket` and `CommonHeader` for framing. +- Implement election as a single class (`OscpElection`) parameterised + by data set type, so the same comparison and state-machine code can + be used for future similar elections if needed. +- Authority and slave behaviours live in one `OscpNode` class with an + internal state machine, not split into separate `OscpAuthority` and + `OscpSlave` classes — every node may become authority at any time. +- The opaque payload is exposed as a `std::span` to the + application layer with no parsing inside OSCP. +- Lifecycle state ([§5.4](#54-lifecycle-states-and-session-only-persistence)) is internal to `OscpNode`; the application + layer observes only a boolean "ACTIVE / not ACTIVE" and a callback + when the payload changes. +- Reassembly buffers ([§6.4](#64-in-protocol-fragmentation)) are owned by `OscpNode`, indexed by + `(authority_uid, transaction_id)`, with a single bounded total-memory + cap shared across all in-flight transactions (the exact cap remains + open per [§11](#11-open-questions)). +- Estimated implementation size: ~1000–1500 lines of C++ including + state machine, election, retransmission, fragmentation/reassembly, + and the message struct definitions.