pools.md

Pools

A pool is a named, weighted group of model lanes that share failover, circuit breaking, and session affinity. Your clients address a pool by name (as the model field), and Busbar decides which backend actually serves each request. Pools are how you turn several providers into one reliable endpoint.

Pools are optional: you can route directly to a single model. But the moment you want weighting, failover, cost-aware routing, or overflow, you reach for a pool.

Client model: "chat" pool: chat weighted · failover gpt-4o via openai weight 8 claude-sonnet via anthropic weight 2 gemini-pro via gemini · tripped, skipped weight 1

The vocabulary

• Pool: a named group of lanes (what a client targets). Owns the selection policy, failover, and affinity.
• Lane: one model at one provider (a models: entry). The unit of concurrency, lifetime budget, and circuit breaking.
• Cell: the breaker state for a specific (pool, lane) pair. A lane that trips in pool A keeps serving in pool B, because each pool has its own cell. See Circuit breaker for the breaker deep-dive.

How selection works

By default a pool uses smooth weighted round-robin (SWRR) over the healthy members: each request goes to the next member by weight, and a tripped, dead, or capacity-exhausted member is skipped with its share redistributed to the rest. If the chosen lane fails before the client has seen a byte, Busbar fails over to the next member, even mid-stream. That is the whole reliability story: weighting for the happy path, automatic failover for the bad one.

Want a different order than weighted? Name a selection strategy — cheapest, fastest, least_busy, usage, or your own ordering hook — as one entry in the pool's hooks: list. That is all of Routing, which owns every strategy, the routing signals, and the ordering-hook contract, with worked examples. The rest of this page is pool structure: members, weights, failover, and affinity.

Config reference

The field-by-field reference — every pool and member field with its type, default, and validation rule — lives in one place: Configuration → pools. This guide stays conceptual so the two never drift.

In short: a pool takes a list of members (each a model lane with an optional weight, context_max, tier, and tags), an optional hooks list (one ordering strategy — weighted/cheapest/fastest/least_busy/usage — plus any gates as inline kind: hook plugin refs), and optional affinity, breaker, failover, and on_exhausted blocks.

Each block with its own guide: Hooks for the selection strategies, the ordering-hook contract, and what a gate receives; Circuit breaker for the per-pool breaker block; and In-flight failover for failover and on_exhausted.

Multi-protocol pools

Multi-protocol pools: members can span different providers and protocols. Busbar translates through its superset IR on cross-protocol hops (see Protocols and translation). A warning is logged at startup for heterogeneous pools because the IR models a common superset: same-protocol requests are byte-exact on the wire (the client sees the upstream's bytes verbatim, with the request side byte-for-byte only when it already names the lane's exact wire model), but cross-protocol hops drop source-only fields that have no analog on the target (e.g. logprobs, n). For pools where all members speak the same protocol there is no field loss and no re-encoding — but responses still pay a per-frame IR decode as a usage side-channel; that cost is not translation overhead in the field-loss sense, but it is not zero either.

Recipes

Weighted split with automatic failover

pools:
  chat:
    members:
      - { model: gpt-4o,        weight: 8 }   # ~80% of traffic
      - { model: claude-sonnet, weight: 2 }   # ~20%
      - { model: gemini-pro,    weight: 1 }   # picks up load when the others trip

Same model, two providers (cross-provider failover)

Run one real model behind two providers. The keys differ; upstream_model carries each provider's own model string. See Configuration.

models:
  sonnet-anthropic: { provider: anthropic,         max_concurrent: 20, upstream_model: claude-3-5-sonnet-20241022 }
  sonnet-bedrock:   { provider: bedrock-us-east-1, max_concurrent: 10, upstream_model: "anthropic.claude-3-5-sonnet-20241022-v2:0" }
pools:
  sonnet:
    members:
      - { model: sonnet-anthropic, weight: 3 }   # primary
      - { model: sonnet-bedrock,   weight: 1 }   # same model, other cloud

Context-length failover

pools:
  long-context:
    members:
      - { model: gpt-4o,        context_max: 128000,  weight: 3 }
      - { model: gemini-15-pro, context_max: 2000000, weight: 1 }   # over-128k requests land here

Sticky sessions

pools:
  agents:
    affinity:
      mode: session
      header_name: x-session-id      # defaults to x-session-id if omitted
    members:
      - { model: gpt-4o,        weight: 1 }
      - { model: claude-sonnet, weight: 1 }

Cost-, latency-, and custom-based routing

Choosing which member serves a request (cheapest, fastest, least busy, or your own kind: hook gate plugin returning an order) is a routing concern, not a pool-shape one: a pool names its selection strategy plus any gates in one hooks: [...] list. Those recipes, with a worked pool example, live in the Hooks guide and the pool-hooks reference.

See the Hooks guide for the full ordering-hook contract and the signals each strategy and gate hook receives, and Circuit breaker / In-flight failover for how the breaker and failover behave once a strategy or gate hook has chosen an order.