> ## Content Index
> Fetch the complete content index at: https://www.ggorantala.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# Kafka UnknownMemberIdException: Why It Happens and How to Fix It
- URL: https://www.ggorantala.dev/kafka-unknownmemberidexception-fix/
- Published: 2026-06-16T14:28:00.000Z
- Updated: 2026-08-29T08:21:55.000Z
- Description: Kafka UnknownMemberIdException — 'The coordinator is not aware of this member' (code 25). Learn the three triggers, the memberId lifecycle, and the exact fix.
- Author: Gopi Gorantala
- Tags: Kafka Consumer Groups, #kafka-consumer-groups, kafka-errors, UnknownMemberIdException, kafka-consumer, consumer-groups, spring-kafka, Java, event-streaming

`UNKNOWN_MEMBER_ID` (error code 25) is the coordinator telling your consumer: *I evicted you, and the member ID you're still using is dead.* It's one of the most misread errors in the consumer group protocol because it surfaces in three mechanically different ways — a heartbeat rejection, a commit failure, and a `SyncGroup` rejection mid-join — and the "increase session.timeout.ms" answer only fixes one of them. This article separates the three, walks the memberId lifecycle that produces them, and gives the fix per trigger.

This is the raw protocol error that often *underlies* what your application-level code reports as `CommitFailedException`. If you're chasing eager-assignor rebalance storms or a cycling-generation loop, see the rebalance-storm article; this one is about the `UNKNOWN_MEMBER_ID` code itself and its distinct paths.

## 1\. The Error

The exact lines you'll paste into a search engine, depending on where it hits:

Heartbeat thread, most common:

```log
[Consumer clientId=consumer-orders-1, groupId=orders] Attempt to heartbeat failed
since member id 'consumer-orders-1-9a3f...' is not valid.

```

```
org.apache.kafka.common.errors.UnknownMemberIdException:
The coordinator is not aware of this member.

```

On the commit path (raw kafka-clients):

```log
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed
since the group has already rebalanced and assigned the partitions to another member.

```

…with the underlying cause in DEBUG:

```log
Offset commit failed on partition orders-3 at offset 148213:
The coordinator is not aware of this member.

```

Mid-join, on the `SyncGroup` response (the loop-y one):

```log
[Consumer clientId=consumer-orders-1, groupId=orders] SyncGroup failed:
The coordinator is not aware of this member. Sent generation was Generation{...}.
Rejoining the group.

```

Environment where it shows up:

- **kafka-clients**: 2.x through 4.x (classic consumer). Wording stabilized post-2.3.
- **Brokers**: any; KRaft or ZooKeeper, single-broker Docker or managed.
- **Setup**: local POCs with slow message handlers, containers under CPU throttling, and long stop-the-world GC pauses in production. Kafka Connect and Kafka Streams workers hit it too — same coordinator protocol underneath.

## 2\. How to Reproduce It

A single broker and a consumer that blocks too long inside the poll loop reproduces the `max.poll.interval.ms` variant deterministically.

`docker-compose.yml` (KRaft, single broker):

```yaml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

```

```bash
docker compose up -d
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --create --topic orders --partitions 3 --replication-factor 1 \
  --bootstrap-server localhost:9092

# feed a few records
docker exec -i kafka /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders --bootstrap-server localhost:9092 <<'EOF'
{"id":1}
{"id":2}
{"id":3}
EOF

```

Consumer that stalls the poll loop past `max.poll.interval.ms` (kafka-clients 3.9.0):

```java
Properties p = new Properties();
p.put("bootstrap.servers", "localhost:9092");
p.put("group.id", "orders");
p.put("key.deserializer", StringDeserializer.class.getName());
p.put("value.deserializer", StringDeserializer.class.getName());
p.put("max.poll.interval.ms", "10000"); // 10s to trigger fast
p.put("max.poll.records", "1");

try (KafkaConsumer<String, String> c = new KafkaConsumer<>(p)) {
    c.subscribe(List.of("orders"));
    while (true) {
        var records = c.poll(Duration.ofMillis(500));
        for (var r : records) {
            Thread.sleep(15_000);   // longer than max.poll.interval.ms
            c.commitSync();         // <-- throws here
        }
    }
}

```

Within \~10s the background thread sends `LeaveGroup`, the coordinator drops the member, and `commitSync()` fails. To reproduce the **session-timeout** variant instead, drop the `sleep`, keep normal processing, and pause the JVM (`kill -STOP <pid>` for 60s, then `kill -CONT`) — the heartbeat thread misses `session.timeout.ms` and the next heartbeat returns `UNKNOWN_MEMBER_ID`.

## 3\. Why It Happens — Surface Level

Every consumer in a group holds two identities the coordinator issues: a **member ID** (a per-session string like `consumer-orders-1-9a3f…`) and a **generation** (the rebalance epoch). The coordinator only honors requests stamped with a member ID it currently knows. `UNKNOWN_MEMBER_ID` means the coordinator has already removed your member from the group's roster, but your client kept using the old ID.

There are exactly three ways to get removed:

1. **Session timeout** — the coordinator didn't receive a heartbeat within `session.timeout.ms` (default 45000ms since Kafka 3.0). The heartbeat thread was starved: long GC pause, frozen/throttled container, or a blocked thread.
2. **`max.poll.interval.ms` breach** — your poll loop took too long between `poll()` calls (default 300000ms). The client *proactively* sends `LeaveGroup`, invalidating its own member ID; the pending commit then fails.
3. **Join/Sync race** — between `JoinGroup` and `SyncGroup`, the member's session expired (typically `poll(Duration.ZERO)` hot loops or heavy startup work), so `SyncGroup` is rejected and the client rejoins from scratch.

## 4\. Why It Happens — Under the Hood

Since KIP-62 (0.10.1), heartbeats run on a **background thread**, decoupled from `poll()`. This split is the whole reason there are two timeouts and two failure modes.

`heartbeat.interval.ms` (default 3000ms) is how often the background thread pings the coordinator. `session.timeout.ms` (45000ms) is how long the coordinator waits before declaring the member dead. The rule of thumb — heartbeat interval ≤ 1/3 of session timeout — exists so a member survives two dropped heartbeats. The broker clamps `session.timeout.ms` to `[group.min.session.timeout.ms, group.max.session.timeout.ms]` \= `[6000, 1800000]`; set it outside that window and `JoinGroup` fails with `InvalidSessionTimeoutException`, not code 25.

`max.poll.interval.ms` is enforced differently. The background thread tracks the wall-clock gap since the last `poll()`. If you exceed it, the thread stops heartbeating *and* sends an explicit `LeaveGroup`. That's a courtesy: rather than let the coordinator wait out the session timeout, the client removes itself immediately so a rebalance can proceed. The side effect is that your member ID is now invalid, so the very next `commitSync()`/`commitAsync()` you issue for the just-processed batch comes back `UNKNOWN_MEMBER_ID` — which the high-level API wraps and rethrows as `CommitFailedException`.

The `SyncGroup` variant is subtler. Group join is two round trips: `JoinGroup` (coordinator picks a leader, collects subscriptions) then `SyncGroup` (leader computes assignments, everyone fetches theirs). If a member is slow to send `SyncGroup` — a `poll(Duration.ZERO)` loop that never lets the coordinator do meaningful work, or blocking initialization on the first poll — its session can expire *inside* the join. The coordinator evicts it and answers `SyncGroup` with code 25\. The client resets its generation and member ID and rejoins, which under load looks like a join loop.

When the client receives `UNKNOWN_MEMBER_ID`, `AbstractCoordinator` resets the member ID to `UNKNOWN_MEMBER_ID` (empty) and the generation to `NO_GENERATION`, then rejoins on the next poll. So the error is frequently **self-healing** — but not free: any offsets not yet committed for the evicted generation are lost, and those records get redelivered to whoever picks up the partitions. Silent duplicates, not a crash, are the real cost.

Under KIP-848 (the new consumer group protocol, GA in Kafka 4.0, `group.protocol=consumer`), member IDs are client-generated UUIDs and heartbeating/assignment is broker-driven, which removes the classic `JoinGroup`/`SyncGroup` race. The failure taxonomy changes, but on the classic protocol — still the default for most deployments — everything above applies.

## 5\. The Fix

Match the fix to the trigger. Blindly bumping `session.timeout.ms` does nothing for a `max.poll.interval.ms` breach.

**Trigger 2 — slow processing (the POC default).** Bound the work per poll, don't just widen the window.

```diff
- props.put("max.poll.records", "500");
- // process each record synchronously, ~50ms each = 25s per batch
+ props.put("max.poll.records", "100");        // cap batch to fit the budget
+ props.put("max.poll.interval.ms", "120000"); // headroom for the slow path

```

If per-record work is genuinely long (external calls, DB writes), hand the batch to a worker pool and pause partitions instead of holding the poll thread — see below.

**Trigger 1 — heartbeat starvation.** This is a GC/scheduling problem, not a config problem. Widen the session only as a stopgap:

```diff
- props.put("session.timeout.ms", "45000");
- props.put("heartbeat.interval.ms", "3000");
+ props.put("session.timeout.ms", "60000");
+ props.put("heartbeat.interval.ms", "20000"); // keep at ~1/3

```

The real fix is removing the pauses: right-size the heap, switch to G1/ZGC for shorter STW, and stop CPU-throttling the container (in Kubernetes, don't set a CPU *limit* far below the request for a latency-sensitive consumer).

**Trigger 3 — join/sync race.** Never poll with `Duration.ZERO` in a loop, and don't block the first poll on heavy initialization. Do warm-up work before `subscribe()`:

```diff
- consumer.subscribe(topics);
- var records = consumer.poll(Duration.ZERO);   // starves the join
+ warmUpCachesAndConnections();                 // before subscribe
+ consumer.subscribe(topics);
+ var records = consumer.poll(Duration.ofMillis(1000));

```

## 6\. Best Practices & The Better Design

The durable fix is to stop coupling slow work to the poll thread. Pull records off, process them on a worker pool, and pause/resume partitions so the poll loop keeps heartbeating and never breaches `max.poll.interval.ms`:

```java
while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    if (!records.isEmpty()) {
        consumer.pause(consumer.assignment());          // stop fetching
        Future<?> f = pool.submit(() -> handle(records));
        while (!f.isDone()) {
            consumer.poll(Duration.ZERO);               // heartbeat only, no new records
        }
        consumer.resume(consumer.assignment());
        consumer.commitSync(offsetsFor(records));
    }
}

```

Here `poll(Duration.ZERO)` is safe *because* partitions are paused — it drives the heartbeat/rebalance machinery without fetching, and no batch is ever held longer than the budget.

In Spring Kafka, don't build this by hand. Set container concurrency to match partitions, keep the listener fast, and let the framework own commit timing:

```java
@Bean
ConcurrentKafkaListenerContainerFactory<String, String> factory(
        ConsumerFactory<String, String> cf) {
    var f = new ConcurrentKafkaListenerContainerFactory<String, String>();
    f.setConsumerFactory(cf);
    f.setConcurrency(3); // == partition count
    f.getContainerProperties().setAckMode(ContainerProperties.AckMode.BATCH);
    return f;
}

@KafkaListener(topics = "orders", groupId = "orders")
public void onMessage(ConsumerRecord<String, String> rec) {
    // keep this fast; offload blocking I/O
}

```

For genuinely long-running handlers, use the container's `pause()`/`resume()` or Spring's async `@KafkaListener` support rather than sleeping on the consumer thread. And pair this with `CooperativeStickyAssignor` and static membership (`group.instance.id`) so the churn caused by a single slow instance doesn't ripple across the group.

## 7\. How to Prevent It Long-Term

- **Alert on the fingerprint.** Watch `rebalance-rate-per-hour` and consumer group generation churn. A climbing generation with flat membership is the "we keep getting evicted" signature. Also alert on consumer lag spikes that coincide with eviction — that's where duplicates hide.
- **Budget the poll loop in CI.** Assert p99 processing-time-per-batch × `max.poll.records` stays comfortably under `max.poll.interval.ms`. Make it a test, not a tribal rule.
- **Measure GC, not just throughput.** Track STW pause p99 against `session.timeout.ms`. If pauses approach a third of the session timeout, the eviction is a matter of time.
- **Kill CPU throttling for consumers.** In Kubernetes, throttled containers freeze the heartbeat thread. Prefer generous requests and avoid tight CPU limits on latency-sensitive consumers.
- **Chaos-test the pause.** `kill -STOP` the consumer JVM for longer than `session.timeout.ms` in a Testcontainers run and assert the group recovers without lost commits under your at-least-once contract.

## 8\. Key Takeaways

- `UNKNOWN_MEMBER_ID` (code 25) means the coordinator already evicted your member and your client is using a dead member ID — it's the protocol error often reported upstream as `CommitFailedException`.
- Three distinct triggers, three distinct fixes: **session timeout** (heartbeat starvation → fix GC/throttling), **`max.poll.interval.ms` breach** (slow processing → cap `max.poll.records`, offload work), and the **join/sync race** (never `poll(Duration.ZERO)` before assignment).
- Heartbeats are decoupled from `poll()` (KIP-62): `session.timeout.ms` guards the heartbeat thread; `max.poll.interval.ms` guards the poll loop. Confusing them is why the wrong knob gets turned.
- The error self-heals via member-ID reset and rejoin, but uncommitted offsets for the evicted generation are lost → silent duplicates, the real damage.
- Design it out: pause/resume with a worker pool (or Spring container-managed commits) so no batch ever outlives the budget, plus cooperative-sticky + static membership to contain the blast radius.