Package fleet is the client for the cc-backend fleet service. With it a
ClusterCockpit service (cc-metric-store, cc-metric-collector, cc-slurm-adapter,
…) does the following:
- registers with cc-backend and gets an instance id,
- receives its configuration from the central configuration tree, with ETag-based polling,
- stays marked alive by sending heartbeats over NATS, or over REST when NATS is unavailable,
- receives discovery rosters of the peers it depends on,
- deregisters on shutdown.
The package also holds the wire definitions shared by client and server: the service type codes, the registration bodies, the roster entry, and the line-protocol encoding of heartbeats and rosters.
For the server side (enabling the service, authoring the configuration tree)
see internal/fleet/README.md in cc-backend.
The application keeps a local fleet section. It only says how to reach
cc-backend and never comes from the fleet service itself.
{
"fleet": {
"url": "https://cc.example.org",
"token": "<jwt with api role>",
"cluster": "fritz",
"heartbeat-interval": "30s",
"config-poll-interval": "60s",
"heartbeat-subject": "cc.fleet.event",
"cache-path": "/var/lib/cc-metric-collector/fleet-config.json"
}
}| Key | Default | Meaning |
|---|---|---|
url |
— (required) | cc-backend base URL |
token |
— | JWT with the api role |
cluster |
— (infra scope) | Cluster of this service; empty registers in infra scope |
hostname |
system hostname | Hostname to register with |
heartbeat-interval |
30s |
Keep it well below cc-backend's stale-after |
config-poll-interval |
60s |
The unchanged case is a header-only round trip |
heartbeat-subject |
— (REST heartbeats) | NATS heartbeat subject, as configured in cc-backend |
discovery-subject-prefix |
cc.fleet.discovery |
Prefix of the roster subjects |
cache-path |
— (no cache) | Last configuration, for starting while cc-backend is down |
timeout |
10s |
Timeout of a single HTTP request |
CC_FLEET_TOKEN, or the file named by CC_FLEET_TOKEN_FILE, overrides
token. ConfigSchema holds the JSON schema of the section.
cfg, err := fleet.ParseConfig(ccconfig.GetPackageConfig("fleet"))
client, err := fleet.New(fleet.Options{
Config: cfg,
ServiceType: fleet.ServiceMetricCollector,
Meta: map[string]string{"version": version}, // broadcast: no secrets
NATS: natsClient.Load, // func() *nats.Client, nil until connected
})
initial, err := client.Bootstrap(ctx, 10*time.Second)
// err == nil: initial.Source is SourceFleet, or SourceNone (nothing authored)
// errors.Is(err, fleet.ErrUnreachable): initial is the cached config (SourceCache) or none
// errors.Is(err, fleet.ErrUnauthorized): token or IP allowlist problem
// Bootstrap logs the outcome itself, so the error needs no further reporting.
applyConfig(initial.Config, true) // nil: run with the local configuration
go client.Run(ctx)
defer client.Close(shutdownCtx) // stops Run, then deregisters
for {
select {
case u := <-client.Configs():
applyConfig(u.Config, false) // only sent when the configuration really changed
case providers := <-client.Rosters():
replaceProviders(providers) // full list, not a delta
case <-ctx.Done():
return
}
}Configs() and Rosters() each hold at most one value. An update that has
not been read yet is replaced by the newer one, so a slow main loop always
sees the latest state and never blocks the client.
Overlay builds the effective configuration from the built-in defaults, the
local configuration file and the fleet configuration, in that order. The
application only states its policy: which keys must stay local (the fleet
section itself, paths, credentials) and which only take effect at startup.
overlay := fleet.Overlay[AppConfig]{
Defaults: defaultConfig, // fresh value per call, no shared maps
Local: localFileContents,
LocalOnly: []string{"fleet", "pid-file"},
StartupOnly: []string{"nats"},
Validate: validate, // optional, may normalize values
}
func applyConfig(blob json.RawMessage, startup bool) {
res, err := overlay.Apply(current, blob, startup)
if err != nil {
log.Printf("rejecting fleet configuration: %v", err) // keep current
return
}
current = res.Config
res.Log() // changed keys, and keys that need a restart
}AppConfig is a struct whose top-level fields are addressed by their JSON
keys. JSON objects decoded into maps merge key by key; arrays and scalars are
replaced. An empty blob yields the local configuration. Fleet keys listed in
LocalOnly are ignored with a warning, unknown keys are logged at debug level,
because the top-level defaults of the configuration tree apply to all service
types. After startup, StartupOnly keys keep their running value and are
reported in Result.RestartRequired.
- Heartbeats go to
heartbeat-subjectwhile the NATS client is connected, and over REST otherwise. A disconnected NATS client never buffers them silently. - Unknown instance. When cc-backend answers
404(the id was rotated or deregistered),Runregisters again and pulls the configuration at once. NATS heartbeats cannot detect this, so the config poll does. - Failures are logged once as a warning, repeated at debug level and followed by an info message on recovery. Registration attempts back off exponentially (1 s up to 5 min), so a rejected token does not hammer cc-backend.
- Change detection compares the configuration body. A cosmetic edit on the server, or a response without an ETag, does not deliver the same configuration again.
- Discovery.
Runsubscribes to<prefix>.<cluster|infra>.<service type>as soon as the NATS provider returns a connected client, and again if it starts returning a different client. - Cache. The cache file is written atomically with mode 0600, because the
configuration may contain secrets. It never contains the instance id and is
ignored if it was written for another service type, scope, cluster or host.
A
204from cc-backend deletes it.
Register, PullConfig, Heartbeat and Deregister are exported for
applications that drive their own loop. Errors unwrap to ErrUnknownInstance
(404), ErrUnauthorized (401/403) or ErrNotRegistered. Other unexpected
statuses are *StatusError.
EncodeHeartbeat, ParseHeartbeat, DecodeHeartbeats, EncodeRoster,
DecodeRoster, DiscoverySubject and Bucket implement the NATS wire format
for both sides.