Skip to content

[server] Add coordinator metrics for rebalance #4055

Description

@morazow

Search before asking

  • I searched in the issues and found nothing similar.

Description

Motivation

Rebalance currently exposes no metrics at all. The only way to observe a rebalance is to poll Admin#listRebalanceProgress or tail coordinator logs. #3965 (PR #3985) improves the Admin API with timestamps and historical listing, but that covers on-demand queries, not continuous monitoring. Operators cannot:

  • Alert when a rebalance is running, stuck, or failing.
  • Chart migration throughput and per-bucket duration on a dashboard.
  • Correlate a rebalance window with write-latency or disk-usage changes on TabletServers.

The coordinator already registers cluster gauges such as bucketCount through CoordinatorMetricGroup (fluss-server/.../metrics/group/CoordinatorMetricGroup.java), and RebalanceManager already tracks all the needed state (inProgressRebalanceTasksQueue, inProgressRebalanceTasks, finishedRebalanceTasks, the in-flight task start timestamp), so this is a wiring task, not a new subsystem.

Proposed solution

Register a rebalance metric set on the coordinator metric group, with names added to MetricNames:

Gauges for the current task:

  • rebalanceInProgress (0 or 1)
  • rebalanceBucketsPending
  • rebalanceBucketsCompleted, rebalanceBucketsFailed, rebalanceBucketsTimedOut
    (split of the finished map by per-bucket status)
  • rebalanceDurationMs (time since the task was registered, 0 when idle)
  • inflightBucketDurationMs (time the current bucket task has been running, 0 when idle)

Counters across tasks (survive task completion, reset on coordinator failover):

  • rebalancesCompletedTotal, rebalancesFailedTotal, rebalancesCanceledTotal

Implementation notes:

  • Expose the values from RebalanceManager via supplier gauges. The backing maps are ConcurrentHashMap and the in-flight fields are volatile, so reads from the metrics reporter thread are safe without touching the coordinator event thread.
  • Register on CoordinatorMetricGroup, following the existing bucketCount pattern.
  • Document the new metrics in website/docs/maintenance/observability/monitor-metrics.md.

Out of scope: per-table or per-bucket metric groups (cardinality), and the Admin API side already covered by #3965.

Willingness to contribute

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions