> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka RebalanceInProgressException: Why It Happens and How to Fix It - URL: https://www.ggorantala.dev/kafka-rebalanceinprogressexception-fix/ - Published: 2026-03-09T14:16:00.000Z - Updated: 2026-08-29T08:26:06.000Z - Description: Kafka RebalanceInProgressException on commitSync/commitAsync with CooperativeStickyAssignor? Here's why it's not CommitFailedException, and the correct fix. - Author: Gopi Gorantala - Tags: #kafka-consumer-groups, kafka-errors, RebalanceInProgressException, kafka-consumer, cooperative-sticky-assignor, spring-kafka, Java, event-streaming If you moved a consumer group to `CooperativeStickyAssignor` and your commits suddenly started throwing `RebalanceInProgressException`, you did not misconfigure anything. You surfaced a control-flow signal that the eager assignor hid from you. This is not `CommitFailedException`. It is retriable, it is expected during incremental rebalances, and if your code or your framework treats it as fatal, you will drop consumers and stall partitions for no reason. This article is about that specific exception: what throws it, why cooperative rebalancing makes it visible, and how to handle it so you stop killing healthy consumers. ## 1\. The Error The exact exception, as it lands in your logs from a `commitSync()` or a synchronous commit inside a rebalance listener: ```text org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation. ``` Wrapped through Spring Kafka's listener container, you will see it inside a broader failure: ```text org.springframework.kafka.KafkaException: Seek to current after exception ... Caused by: org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. ``` The wire-level identity: error code `27`, `REBALANCE_IN_PROGRESS`. The Java class is `org.apache.kafka.common.errors.RebalanceInProgressException`, and critically it extends `ApiException` — **not** `RetriableException`, and **not** `RetriableCommitFailedException`. Where it shows up: - **Kafka version:** any release since `kafka-clients` 2.4 (KIP-429, cooperative rebalancing). It becomes common the moment you set `partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor`, which is the default in Kafka 3.x reference configs and the assignor everyone migrates to after a rebalance-storm incident. - **Setup:** local Docker POCs, Kubernetes StatefulSets, managed Kafka — assignor-independent of infra. The trigger is *incremental* rebalancing plus a commit that races the rebalance. - **Clients affected:** raw `kafka-clients`, Spring Kafka, reactor-kafka, and anything that only catches `RetriableCommitFailedException` on commit. ## 2\. How to Reproduce It You need two consumers in one group using the cooperative assignor, and a commit that fires while a rebalance is in flight. Start a local KRaft broker: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.0 container_name: kafka ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 ``` ```bash docker compose up -d docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --create --topic orders --partitions 6 --replication-factor 1 ``` A consumer that commits synchronously per batch, cooperative assignor: ```java // kafka-clients 3.9.0 Properties p = new Properties(); p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); p.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-svc"); p.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, "org.apache.kafka.clients.consumer.CooperativeStickyAssignor"); p.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); try (KafkaConsumer c = new KafkaConsumer<>(p)) { c.subscribe(List.of("orders")); while (true) { ConsumerRecords records = c.poll(Duration.ofMillis(500)); // ... process ... c.commitSync(); // <-- throws RebalanceInProgressException during a join } } ``` Now force the race. Run the consumer above, wait for it to get its assignment, then start a **second** instance of the same process. The second consumer's `JoinGroup` triggers a cooperative rebalance. During the revocation round trip, the first consumer's `commitSync()` fires against a generation that is mid-rebalance, and the broker's `GroupCoordinator` rejects it: ```text org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation. ``` You can widen the window deterministically by slowing the processing loop (`Thread.sleep(2000)` before `commitSync()`) and bouncing the second instance a few times. With `RangeAssignor` or `RoundRobinAssignor` (eager), the same scenario throws `CommitFailedException` instead — that difference is the whole point of this article. ## 3\. Why It Happens — Surface Level Your consumer tried to commit offsets while the group was in the middle of a rebalance. The coordinator can't accept a commit for a generation that is being torn down and rebuilt, so it returns `REBALANCE_IN_PROGRESS`. The exception message tells you the remedy verbatim: call `poll()` to drive the rebalance to completion, then retry the commit. This is fundamentally a *timing* condition, not a *misconfiguration* condition. It says "not now," not "you lost your partitions." That distinction is everything. ## 4\. Why It Happens — Under the Hood ### The two exceptions are not interchangeable Under the **eager** protocol (`RangeAssignor`, `RoundRobinAssignor`, `StickyAssignor`), every rebalance revokes *all* partitions from *all* members first, then reassigns. A member that took too long between polls gets kicked, and when it finally commits it hears: the group already rebalanced and gave your partitions to someone else. That is `CommitFailedException` — the assignment is *gone*, the commit is meaningless, and the only correct response is to stop and reprocess from the new position. It is fatal by design. Under the **cooperative** protocol (`CooperativeStickyAssignor`, KIP-429), rebalances are incremental. Members keep the partitions they will retain and only revoke the specific partitions being moved. A rebalance is a normal, frequent, low-cost event — and it can overlap with a commit that is otherwise perfectly valid. When it does, the coordinator returns `REBALANCE_IN_PROGRESS` (code 27) rather than declaring your membership dead. Your generation is still alive; you are simply mid-transition. That is `RebalanceInProgressException` — **retriable**. ### The commit path When you call `commitSync()`, the consumer sends an `OffsetCommit` request tagged with your current `generationId` and `memberId`. The `GroupCoordinator` on the broker validates that generation against the group's state machine (`Stable`, `PreparingRebalance`, `CompletingRebalance`, `Empty`). If the group is in `PreparingRebalance` or `CompletingRebalance` — i.e., a `JoinGroup`/`SyncGroup` round is underway — the commit for the in-flight generation cannot be durably recorded, so the coordinator answers `REBALANCE_IN_PROGRESS`. The consumer surfaces it as `RebalanceInProgressException`. The message's advice — "call poll()" — is literal. `poll()` is the only place the consumer's heartbeat/coordinator thread advances the rebalance: it sends the `JoinGroup`, receives the assignment via `SyncGroup`, runs your `ConsumerRebalanceListener` callbacks, and returns the group to `Stable`. Only after that will the commit for the new generation succeed. ### The trap: `RetriableCommitFailedException` Here is the sharp edge that breaks real systems. `RetriableCommitFailedException` is the exception class that async-commit callbacks and many frameworks were written to catch-and-retry. `RebalanceInProgressException` is **not** a subclass of it. So code that looks like this: ```java if (exception instanceof RetriableCommitFailedException) { retry(); // never reached for RebalanceInProgressException } else { giveUp(); // <-- healthy consumer thrown away here } ``` ...treats a benign, retriable rebalance signal as fatal. reactor-kafka and several homegrown consumer wrappers historically had exactly this bug: a consumer commits during a cooperative rebalance, gets `RebalanceInProgressException`, the wrapper doesn't recognize it as retriable, tears the consumer down, which triggers *another* rebalance — a self-inflicted churn loop that looks like a rebalance storm but is really just mishandled commit semantics. ## 5\. The Fix ### Fix A — Raw kafka-clients: retry after poll() Do exactly what the message says. Catch it, poll to complete the rebalance, retry the commit. ```diff while (true) { ConsumerRecords records = c.poll(Duration.ofMillis(500)); process(records); - c.commitSync(); + try { + c.commitSync(); + } catch (RebalanceInProgressException e) { + // benign: a rebalance is in flight. The next poll() drives it to + // completion; offsets will be committed on the next successful cycle + // (or in onPartitionsRevoked for partitions we're giving up). + log.info("Commit deferred, rebalance in progress: {}", e.getMessage()); + } catch (CommitFailedException e) { + // fatal: assignment already reassigned. Do NOT blindly retry. + log.warn("Assignment lost, will reprocess from new position", e); + } } ``` Note the two-tier catch. `RebalanceInProgressException` is recoverable — swallow, keep looping, the next `poll()` heals it. `CommitFailedException` means your partitions are already someone else's; retrying the same commit is wrong. ### Fix B — Commit the retained partitions in the rebalance listener The cleanest place to commit under cooperative rebalancing is in `onPartitionsRevoked`, which fires only for the partitions actually being moved. Commit there, and your steady-state loop can commit less aggressively: ```java c.subscribe(List.of("orders"), new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection revoked) { // called during poll(); safe to commit the offsets we own for the // partitions we're handing off c.commitSync(offsetsFor(revoked)); } @Override public void onPartitionsAssigned(Collection assigned) { } @Override public void onPartitionsLost(Collection lost) { // cooperative: do NOT commit here — ownership already gone } }); ``` ### Fix C — Spring Kafka: let the container classify it Spring Kafka's `DefaultErrorHandler` already knows `RebalanceInProgressException` is not something to seek-and-retry, but you should make the intent explicit and not treat it as a record failure. Use container-managed commits (`AckMode.BATCH` or `RECORD`) so Spring commits at safe points, and add it to the non-retryable set so it never counts against a record's retry budget: ```java @Bean DefaultErrorHandler errorHandler(KafkaTemplate template) { var recoverer = new DeadLetterPublishingRecoverer(template); var handler = new DefaultErrorHandler(recoverer, new ExponentialBackOffWithMaxRetries(5)); // rebalance-in-progress is transient control flow, not a bad record handler.addNotRetryableExceptions(RebalanceInProgressException.class); return handler; } ``` ```java @Bean ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory( ConsumerFactory cf, DefaultErrorHandler handler) { var factory = new ConcurrentKafkaListenerContainerFactory(); factory.setConsumerFactory(cf); factory.getContainerProperties().setAckMode(AckMode.BATCH); factory.setCommonErrorHandler(handler); return factory; } ``` Which fix to use: **Fix A** for low-level consumers you own; **Fix B** whenever you do manual commits and care about exactly-where offsets land during moves; **Fix C** for any Spring Boot service — let the container own commit timing instead of hand-committing on the listener thread. ## 6\. Best Practices & The Better Design The better design is to stop hand-committing on the hot path and let commit timing be driven by the poll loop, so commits never race a rebalance you didn't coordinate. ```java // The right way: manual-immediate acks handed back to the container, // cooperative assignor, static membership to suppress needless rebalances. @KafkaListener(topics = "orders", groupId = "orders-svc") public void onMessage(ConsumerRecord rec, Acknowledgment ack) { process(rec); ack.acknowledge(); // container commits at a safe point, not mid-rebalance } ``` ```properties # consumer config partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor group.instance.id=orders-svc-${POD_ORDINAL} # KIP-345 static membership session.timeout.ms=45000 heartbeat.interval.ms=3000 max.poll.interval.ms=300000 enable.auto.commit=false ``` Three principles behind that config: - **Cooperative assignor everywhere in the group.** Mixed assignors force an eager fallback and reintroduce stop-the-world rebalances. Roll every member to `CooperativeStickyAssignor` before you rely on incremental behavior. - **Static membership (`group.instance.id`)** parks a member's assignment across brief restarts, so a rolling deploy doesn't trigger a rebalance at all — no rebalance, no `RebalanceInProgressException`. Derive the ID from a StatefulSet ordinal or pod name; never hardcode it (that path leads to `FencedInstanceIdException`). - **Commit in the listener/container, not in a bare loop.** Framework-managed commits happen at points where the consumer isn't mid-join, which structurally avoids the race. If you genuinely need manual commits, treat `RebalanceInProgressException` as a normal continue-and-retry signal, and reserve fatal handling for `CommitFailedException`. ## 7\. How to Prevent It Long-Term Prevention is mostly about reducing *rebalance frequency* and making sure the exception is classified correctly in code review. - **Alert on rebalance rate, not just lag.** Track the consumer `rebalance-rate-per-hour` and `rebalance-total` JMX metrics. A cooperative group should rebalance on deploys and scaling — not continuously. A rising `RebalanceInProgressException` count with flat membership means a commit-during-rebalance loop, usually a mishandled exception. - **Ban the fatal-classification bug in CI.** Add a check (ArchUnit, or a simple grep in review) that any `catch` on commit which special-cases `RetriableCommitFailedException` also accounts for `RebalanceInProgressException`. This is the single most common way the error becomes an outage. - **Standardize commit strategy per team.** Pick container-managed commits (Spring `AckMode`) or a documented manual pattern — don't let each service invent its own hand-commit loop. - **Load-test at real partition and instance counts.** The race only appears under concurrent membership change. A two-instance Testcontainers test that starts a second consumer mid-processing and asserts no consumer dies reproduces it deterministically; run it in CI. - **Keep `max.poll.interval.ms` honest.** Slow processing that breaches the poll interval causes *extra* rebalances, which multiply your exposure to this race. Bound per-batch work with `max.poll.records`. ## 8\. Key Takeaways - `RebalanceInProgressException` (code 27) means "a rebalance is in flight, retry after `poll()`" — it is **retriable** and expected under `CooperativeStickyAssignor`. It is **not** `CommitFailedException`, which means "your assignment is gone" and is fatal. - It is **not** a subclass of `RetriableCommitFailedException`. Code that only retries `RetriableCommitFailedException` will wrongly kill healthy consumers and create a self-inflicted churn loop. - Fix: catch it, `poll()` to complete the rebalance, retry the commit — or commit retained partitions in `onPartitionsRevoked`. In Spring, use container-managed acks and add it to `addNotRetryableExceptions`. - Design it away: cooperative assignor across the whole group, static membership to suppress needless rebalances, and container-managed commit timing instead of a bare `commitSync()` on the hot path. - Monitor `rebalance-rate-per-hour`; a climbing rebalance rate with stable membership is the fingerprint of this exception being mishandled.