Skip to content

Cooperative rebalancing consumer hangs on disconnect, rebalanceProtocol() returns NONE #1144

Open

Description

@kathrynkodama

Hi there, I believe this is a bug in the node-rdkafka library. When we use cooperative rebalancing, on shutdown some consumers return the rebalance protocol as NONE rather than COOPERATIVE.

Environment Information

  • OS [e.g. Mac, Arch, Windows 10]:
  • Node Version [e.g. 8.2.1]: 22.22.0
  • NPM Version [e.g. 5.4.2]: 10.9.4
  • C++ Toolchain [e.g. Visual Studio, llvm, g++]:
  • node-rdkafka version [e.g. 2.3.3]: 3.6.0

Steps to Reproduce

  1. Create a KafkaConsumer with partition.assignment.strategy: 'cooperative-sticky' and rebalance_cb: true
  2. Connect the consumer and wait for it to join a consumer group and receive partition assignments
  3. Call consumer.disconnect() (or stream.destroy() on a KafkaConsumerStream)
  4. Observe that the final revoke rebalance callback fires, but rebalanceProtocol() returns 'NONE' instead of 'COOPERATIVE'
  5. It looks like the built-in handler calls unassign() instead of incrementalUnassign(), which throws inside librdkafka. The error is silently swallowed because isConnected() is also false at this point. disconnect() never completes.

node-rdkafka Configuration Settings

'partition.assignment.strategy': 'cooperative-sticky',
rebalance_cb: true,

Additional context

We observe this happening intermittently, when we have multiple consumers with a large number of partitions. When rebalancing happens, only some consumers return 'NONE' instead of 'COOPERATIVE'.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions