Search before asking
Description
Motivation
Rebalance currently exposes no metrics at all. The only way to observe a rebalance is to poll Admin#listRebalanceProgress or tail coordinator logs. #3965 (PR #3985) improves the Admin API with timestamps and historical listing, but that covers on-demand queries, not continuous monitoring. Operators cannot:
- Alert when a rebalance is running, stuck, or failing.
- Chart migration throughput and per-bucket duration on a dashboard.
- Correlate a rebalance window with write-latency or disk-usage changes on TabletServers.
The coordinator already registers cluster gauges such as bucketCount through CoordinatorMetricGroup (fluss-server/.../metrics/group/CoordinatorMetricGroup.java), and RebalanceManager already tracks all the needed state (inProgressRebalanceTasksQueue, inProgressRebalanceTasks, finishedRebalanceTasks, the in-flight task start timestamp), so this is a wiring task, not a new subsystem.
Proposed solution
Register a rebalance metric set on the coordinator metric group, with names added to MetricNames:
Gauges for the current task:
rebalanceInProgress (0 or 1)
rebalanceBucketsPending
rebalanceBucketsCompleted, rebalanceBucketsFailed, rebalanceBucketsTimedOut
(split of the finished map by per-bucket status)
rebalanceDurationMs (time since the task was registered, 0 when idle)
inflightBucketDurationMs (time the current bucket task has been running, 0 when idle)
Counters across tasks (survive task completion, reset on coordinator failover):
rebalancesCompletedTotal, rebalancesFailedTotal, rebalancesCanceledTotal
Implementation notes:
- Expose the values from
RebalanceManager via supplier gauges. The backing maps are ConcurrentHashMap and the in-flight fields are volatile, so reads from the metrics reporter thread are safe without touching the coordinator event thread.
- Register on
CoordinatorMetricGroup, following the existing bucketCount pattern.
- Document the new metrics in
website/docs/maintenance/observability/monitor-metrics.md.
Out of scope: per-table or per-bucket metric groups (cardinality), and the Admin API side already covered by #3965.
Willingness to contribute
Search before asking
Description
Motivation
Rebalance currently exposes no metrics at all. The only way to observe a rebalance is to poll
Admin#listRebalanceProgressor tail coordinator logs. #3965 (PR #3985) improves the Admin API with timestamps and historical listing, but that covers on-demand queries, not continuous monitoring. Operators cannot:The coordinator already registers cluster gauges such as
bucketCountthroughCoordinatorMetricGroup(fluss-server/.../metrics/group/CoordinatorMetricGroup.java), andRebalanceManageralready tracks all the needed state (inProgressRebalanceTasksQueue,inProgressRebalanceTasks,finishedRebalanceTasks, the in-flight task start timestamp), so this is a wiring task, not a new subsystem.Proposed solution
Register a rebalance metric set on the coordinator metric group, with names added to
MetricNames:Gauges for the current task:
rebalanceInProgress(0 or 1)rebalanceBucketsPendingrebalanceBucketsCompleted,rebalanceBucketsFailed,rebalanceBucketsTimedOut(split of the finished map by per-bucket status)
rebalanceDurationMs(time since the task was registered, 0 when idle)inflightBucketDurationMs(time the current bucket task has been running, 0 when idle)Counters across tasks (survive task completion, reset on coordinator failover):
rebalancesCompletedTotal,rebalancesFailedTotal,rebalancesCanceledTotalImplementation notes:
RebalanceManagervia supplier gauges. The backing maps areConcurrentHashMapand the in-flight fields are volatile, so reads from the metrics reporter thread are safe without touching the coordinator event thread.CoordinatorMetricGroup, following the existingbucketCountpattern.website/docs/maintenance/observability/monitor-metrics.md.Out of scope: per-table or per-bucket metric groups (cardinality), and the Admin API side already covered by #3965.
Willingness to contribute