Kafka RebalanceInProgressException: Why It Happens and How to Fix It

If you moved a consumer group to CooperativeStickyAssignor and your commits suddenly started throwing RebalanceInProgressException, you did not misconfigure anything. You surfaced a control-flow signal that the eager assignor hid from you. This is not CommitFailedException. It is retriable, it is expected during incremental rebalances, and if your code or your framework treats it as fatal, you will drop consumers and stall partitions for no reason.

This article is about that specific exception: what throws it, why cooperative rebalancing makes it visible, and how to handle it so you stop killing healthy consumers.

1. The Error

The exact exception, as it lands in your logs from a commitSync() or a synchronous commit inside a rebalance listener:

org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit
cannot be completed since the consumer is undergoing a rebalance for auto
partition assignment. You can try completing the rebalance by calling poll()
and then retry the operation.

Wrapped through Spring Kafka's listener container, you will see it inside a broader failure:

org.springframework.kafka.KafkaException: Seek to current after exception
  ... 
Caused by: org.apache.kafka.common.errors.RebalanceInProgressException:
  Offset commit cannot be completed since the consumer is undergoing a
  rebalance for auto partition assignment.

The wire-level identity: error code 27, REBALANCE_IN_PROGRESS. The Java class is org.apache.kafka.common.errors.RebalanceInProgressException, and critically it extends ApiException — not RetriableException, and not RetriableCommitFailedException.

Where it shows up:

• Kafka version: any release since kafka-clients 2.4 (KIP-429, cooperative rebalancing). It becomes common the moment you set partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor, which is the default in Kafka 3.x reference configs and the assignor everyone migrates to after a rebalance-storm incident.
• Setup: local Docker POCs, Kubernetes StatefulSets, managed Kafka — assignor-independent of infra. The trigger is incremental rebalancing plus a commit that races the rebalance.
• Clients affected: raw kafka-clients, Spring Kafka, reactor-kafka, and anything that only catches RetriableCommitFailedException on commit.

2. How to Reproduce It

You need two consumers in one group using the cooperative assignor, and a commit that fires while a rebalance is in flight. Start a local KRaft broker:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

docker compose up -d
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 6 --replication-factor 1

A consumer that commits synchronously per batch, cooperative assignor:

// kafka-clients 3.9.0
Properties p = new Properties();
p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-svc");
p.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
      "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
p.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

try (KafkaConsumer<String, String> c = new KafkaConsumer<>(p)) {
    c.subscribe(List.of("orders"));
    while (true) {
        ConsumerRecords<String, String> records = c.poll(Duration.ofMillis(500));
        // ... process ...
        c.commitSync();   // <-- throws RebalanceInProgressException during a join
    }
}

Now force the race. Run the consumer above, wait for it to get its assignment, then start a second instance of the same process. The second consumer's JoinGroup triggers a cooperative rebalance. During the revocation round trip, the first consumer's commitSync() fires against a generation that is mid-rebalance, and the broker's GroupCoordinator rejects it:

org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit
cannot be completed since the consumer is undergoing a rebalance for auto
partition assignment. You can try completing the rebalance by calling poll()
and then retry the operation.

You can widen the window deterministically by slowing the processing loop (Thread.sleep(2000) before commitSync()) and bouncing the second instance a few times. With RangeAssignor or RoundRobinAssignor (eager), the same scenario throws CommitFailedException instead — that difference is the whole point of this article.

3. Why It Happens — Surface Level

Your consumer tried to commit offsets while the group was in the middle of a rebalance. The coordinator can't accept a commit for a generation that is being torn down and rebuilt, so it returns REBALANCE_IN_PROGRESS. The exception message tells you the remedy verbatim: call poll() to drive the rebalance to completion, then retry the commit.

This is fundamentally a timing condition, not a misconfiguration condition. It says "not now," not "you lost your partitions." That distinction is everything.

4. Why It Happens — Under the Hood

The two exceptions are not interchangeable

Under the eager protocol (RangeAssignor, RoundRobinAssignor, StickyAssignor), every rebalance revokes all partitions from all members first, then reassigns. A member that took too long between polls gets kicked, and when it finally commits it hears: the group already rebalanced and gave your partitions to someone else. That is CommitFailedException — the assignment is gone, the commit is meaningless, and the only correct response is to stop and reprocess from the new position. It is fatal by design.

Under the cooperative protocol (CooperativeStickyAssignor, KIP-429), rebalances are incremental. Members keep the partitions they will retain and only revoke the specific partitions being moved. A rebalance is a normal, frequent, low-cost event — and it can overlap with a commit that is otherwise perfectly valid. When it does, the coordinator returns REBALANCE_IN_PROGRESS (code 27) rather than declaring your membership dead. Your generation is still alive; you are simply mid-transition. That is RebalanceInProgressException — retriable.

The commit path

When you call commitSync(), the consumer sends an OffsetCommit request tagged with your current generationId and memberId. The GroupCoordinator on the broker validates that generation against the group's state machine (Stable, PreparingRebalance, CompletingRebalance, Empty). If the group is in PreparingRebalance or CompletingRebalance — i.e., a JoinGroup/SyncGroup round is underway — the commit for the in-flight generation cannot be durably recorded, so the coordinator answers REBALANCE_IN_PROGRESS. The consumer surfaces it as RebalanceInProgressException.

The message's advice — "call poll()" — is literal. poll() is the only place the consumer's heartbeat/coordinator thread advances the rebalance: it sends the JoinGroup, receives the assignment via SyncGroup, runs your ConsumerRebalanceListener callbacks, and returns the group to Stable. Only after that will the commit for the new generation succeed.

The trap: RetriableCommitFailedException

Here is the sharp edge that breaks real systems. RetriableCommitFailedException is the exception class that async-commit callbacks and many frameworks were written to catch-and-retry. RebalanceInProgressException is not a subclass of it. So code that looks like this:

if (exception instanceof RetriableCommitFailedException) {
    retry();      // never reached for RebalanceInProgressException
} else {
    giveUp();     // <-- healthy consumer thrown away here
}

...treats a benign, retriable rebalance signal as fatal. reactor-kafka and several homegrown consumer wrappers historically had exactly this bug: a consumer commits during a cooperative rebalance, gets RebalanceInProgressException, the wrapper doesn't recognize it as retriable, tears the consumer down, which triggers another rebalance — a self-inflicted churn loop that looks like a rebalance storm but is really just mishandled commit semantics.

5. The Fix

Fix A — Raw kafka-clients: retry after poll()

Do exactly what the message says. Catch it, poll to complete the rebalance, retry the commit.

  while (true) {
      ConsumerRecords<String, String> records = c.poll(Duration.ofMillis(500));
      process(records);
-     c.commitSync();
+     try {
+         c.commitSync();
+     } catch (RebalanceInProgressException e) {
+         // benign: a rebalance is in flight. The next poll() drives it to
+         // completion; offsets will be committed on the next successful cycle
+         // (or in onPartitionsRevoked for partitions we're giving up).
+         log.info("Commit deferred, rebalance in progress: {}", e.getMessage());
+     } catch (CommitFailedException e) {
+         // fatal: assignment already reassigned. Do NOT blindly retry.
+         log.warn("Assignment lost, will reprocess from new position", e);
+     }
  }

Note the two-tier catch. RebalanceInProgressException is recoverable — swallow, keep looping, the next poll() heals it. CommitFailedException means your partitions are already someone else's; retrying the same commit is wrong.

Fix B — Commit the retained partitions in the rebalance listener

The cleanest place to commit under cooperative rebalancing is in onPartitionsRevoked, which fires only for the partitions actually being moved. Commit there, and your steady-state loop can commit less aggressively:

c.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> revoked) {
        // called during poll(); safe to commit the offsets we own for the
        // partitions we're handing off
        c.commitSync(offsetsFor(revoked));
    }
    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> assigned) { }
    @Override
    public void onPartitionsLost(Collection<TopicPartition> lost) {
        // cooperative: do NOT commit here — ownership already gone
    }
});

Fix C — Spring Kafka: let the container classify it

Spring Kafka's DefaultErrorHandler already knows RebalanceInProgressException is not something to seek-and-retry, but you should make the intent explicit and not treat it as a record failure. Use container-managed commits (AckMode.BATCH or RECORD) so Spring commits at safe points, and add it to the non-retryable set so it never counts against a record's retry budget:

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    var handler = new DefaultErrorHandler(recoverer,
            new ExponentialBackOffWithMaxRetries(5));
    // rebalance-in-progress is transient control flow, not a bad record
    handler.addNotRetryableExceptions(RebalanceInProgressException.class);
    return handler;
}

@Bean
ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
        ConsumerFactory<String, String> cf, DefaultErrorHandler handler) {
    var factory = new ConcurrentKafkaListenerContainerFactory<String, String>();
    factory.setConsumerFactory(cf);
    factory.getContainerProperties().setAckMode(AckMode.BATCH);
    factory.setCommonErrorHandler(handler);
    return factory;
}

Which fix to use: Fix A for low-level consumers you own; Fix B whenever you do manual commits and care about exactly-where offsets land during moves; Fix C for any Spring Boot service — let the container own commit timing instead of hand-committing on the listener thread.

6. Best Practices & The Better Design

The better design is to stop hand-committing on the hot path and let commit timing be driven by the poll loop, so commits never race a rebalance you didn't coordinate.

// The right way: manual-immediate acks handed back to the container,
// cooperative assignor, static membership to suppress needless rebalances.
@KafkaListener(topics = "orders", groupId = "orders-svc")
public void onMessage(ConsumerRecord<String, String> rec, Acknowledgment ack) {
    process(rec);
    ack.acknowledge();   // container commits at a safe point, not mid-rebalance
}

# consumer config
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id=orders-svc-${POD_ORDINAL}   # KIP-345 static membership
session.timeout.ms=45000
heartbeat.interval.ms=3000
max.poll.interval.ms=300000
enable.auto.commit=false

Three principles behind that config:

• Cooperative assignor everywhere in the group. Mixed assignors force an eager fallback and reintroduce stop-the-world rebalances. Roll every member to CooperativeStickyAssignor before you rely on incremental behavior.
• Static membership (group.instance.id) parks a member's assignment across brief restarts, so a rolling deploy doesn't trigger a rebalance at all — no rebalance, no RebalanceInProgressException. Derive the ID from a StatefulSet ordinal or pod name; never hardcode it (that path leads to FencedInstanceIdException).
• Commit in the listener/container, not in a bare loop. Framework-managed commits happen at points where the consumer isn't mid-join, which structurally avoids the race.

If you genuinely need manual commits, treat RebalanceInProgressException as a normal continue-and-retry signal, and reserve fatal handling for CommitFailedException.

7. How to Prevent It Long-Term

Prevention is mostly about reducing rebalance frequency and making sure the exception is classified correctly in code review.

• Alert on rebalance rate, not just lag. Track the consumer rebalance-rate-per-hour and rebalance-total JMX metrics. A cooperative group should rebalance on deploys and scaling — not continuously. A rising RebalanceInProgressException count with flat membership means a commit-during-rebalance loop, usually a mishandled exception.
• Ban the fatal-classification bug in CI. Add a check (ArchUnit, or a simple grep in review) that any catch on commit which special-cases RetriableCommitFailedException also accounts for RebalanceInProgressException. This is the single most common way the error becomes an outage.
• Standardize commit strategy per team. Pick container-managed commits (Spring AckMode) or a documented manual pattern — don't let each service invent its own hand-commit loop.
• Load-test at real partition and instance counts. The race only appears under concurrent membership change. A two-instance Testcontainers test that starts a second consumer mid-processing and asserts no consumer dies reproduces it deterministically; run it in CI.
• Keep max.poll.interval.ms honest. Slow processing that breaches the poll interval causes extra rebalances, which multiply your exposure to this race. Bound per-batch work with max.poll.records.

8. Key Takeaways

• RebalanceInProgressException (code 27) means "a rebalance is in flight, retry after poll()" — it is retriable and expected under CooperativeStickyAssignor. It is not CommitFailedException, which means "your assignment is gone" and is fatal.
• It is not a subclass of RetriableCommitFailedException. Code that only retries RetriableCommitFailedException will wrongly kill healthy consumers and create a self-inflicted churn loop.
• Fix: catch it, poll() to complete the rebalance, retry the commit — or commit retained partitions in onPartitionsRevoked. In Spring, use container-managed acks and add it to addNotRetryableExceptions.
• Design it away: cooperative assignor across the whole group, static membership to suppress needless rebalances, and container-managed commit timing instead of a bare commitSync() on the hot path.
• Monitor rebalance-rate-per-hour; a climbing rebalance rate with stable membership is the fingerprint of this exception being mishandled.