> ## Content Index
> Fetch the complete content index at: https://www.ggorantala.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# Kafka: Marking the Coordinator Dead — Causes and Fix
- URL: https://www.ggorantala.dev/kafka-marking-the-coordinator-dead-fix/
- Published: 2025-10-15T10:02:00.000Z
- Updated: 2026-08-29T08:27:32.000Z
- Description: Kafka consumer stuck logging 'Marking the coordinator dead' / 'Group coordinator is unavailable or invalid, will attempt rediscovery'. Here's the real cause and the fix.
- Author: Gopi Gorantala
- Tags: #kafka-consumer-groups, kafka-errors, kafka-consumer, consumer-groups, group-coordinator, kafka-docker, advertised-listeners, Java

Your consumer connected fine, fetched metadata, maybe even produced. Then it wedged in a loop, spraying this every few seconds and never consuming a record:

```text
[Consumer clientId=order-consumer-0, groupId=order-service] Group coordinator
kafka:29092 (id: 2147483644 rack: null) is unavailable or invalid due to cause:
coordinator unavailable. isDisconnected: true. Rediscovery will be attempted.

```

On older `kafka-clients` (pre-2.0) the same event reads:

```text
[Consumer clientId=order-consumer-0, groupId=order-service] Marking the coordinator
kafka:29092 (id: 2147483644 rack: null) dead for group order-service

```

Followed, endlessly, by:

```text
[Consumer clientId=order-consumer-0, groupId=order-service] Discovered group
coordinator kafka:29092 (id: 2147483644 rack: null)

```

Discover → dead → discover → dead. The consumer never joins the group, never gets an assignment, `poll()` returns empty forever, and lag climbs. This is one of the most-pasted lines in Kafka support history, and 90% of the time in a POC it is **not** a broker problem — it's a networking problem hiding behind a friendly-looking INFO log.

## 1\. The Error

The message originates in `AbstractCoordinator` (the base class behind `ConsumerCoordinator`) in `kafka-clients`. It is logged at **INFO**, which is why people miss it — nothing throws, nothing crashes, the app just does nothing.

- **Client:** `org.apache.kafka:kafka-clients` 3.x (wording confirmed on 3.6–3.9; identical mechanics on 4.0). Spring Kafka 3.x wraps the same client.
- **Broker:** Kafka 3.x / 4.0, KRaft mode.
- **Where it bites:** Docker Compose and Kubernetes POCs, and any multi-listener cluster where the client can reach the *bootstrap* endpoint but not the *coordinator's advertised* endpoint. Also real production incidents when the coordinator broker dies or `__consumer_offsets` has no leader.

The tell that separates this from a plain bootstrap failure (`Connection to node -1 could not be established`): here the client **already has cluster metadata** and has resolved a specific coordinator node with a synthetic id like `2147483644`. It is one hop further along than a bootstrap failure.

## 2\. How to Reproduce It

The cleanest reproduction is the Docker listener trap: advertise a hostname the host machine can't resolve. Bootstrap discovery works because you connect by IP/port, but the coordinator gets advertised under an internal name.

```yaml
# docker-compose.yml — Kafka 3.9 KRaft, deliberately broken listeners
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      # BUG: advertises the internal container name to host clients
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      # single broker: shrink internal topic RF so it can actually get a leader
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

```

Consumer, run from the **host** (not inside a container):

```java
// kafka-clients 3.9.0
Properties p = new Properties();
p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");
p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
p.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

try (KafkaConsumer<String, String> c = new KafkaConsumer<>(p)) {
    c.subscribe(List.of("orders"));
    while (true) {
        var records = c.poll(Duration.ofSeconds(1));
        System.out.println("got " + records.count());
    }
}

```

`localhost:9092` works for bootstrap — the port is published — so metadata loads. But the coordinator comes back advertised as `kafka:29092` (or `kafka:9092`), which does not resolve from your laptop. The consumer discovers the coordinator, fails to open a connection to it, marks it dead, and loops. Turn on `log4j` at `INFO` for `org.apache.kafka.clients.consumer.internals` and you'll see the discover/dead cycle immediately.

To confirm which `__consumer_offsets` partition (and therefore which broker) owns your group:

```bash
docker exec -it kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group order-service --state

```

The `COORDINATOR` column shows the host:port the group is pinned to. If that host:port is unreachable from where your client runs, you've found it.

## 3\. Why It Happens — Surface Level

A consumer group is managed by exactly one broker: the **group coordinator**. To find it, the client sends a `FindCoordinator` request and gets back the broker that leads the `__consumer_offsets` partition owning your group. The client then opens a **separate connection** to that broker — using the address the broker advertises — for all group traffic: `JoinGroup`, `SyncGroup`, `Heartbeat`, `OffsetCommit`, `OffsetFetch`.

"Marking the coordinator dead" means that separate coordinator connection failed, or the coordinator returned a "not me / not ready" error. The client drops its cached coordinator and re-runs `FindCoordinator`. If the underlying cause is stable (wrong advertised address, missing internal topic, dead broker), rediscovery returns the same bad answer and you loop.

## 4\. Why It Happens — Under the Hood

Three mechanically distinct failure modes produce the identical log line.

**(A) Coordinator advertised on an address the client can't reach.** This is the Docker/K8s classic. `FindCoordinator` doesn't return an abstract "broker 1" — it returns a `Node` built from that broker's `advertised.listeners` for the listener the client is using. Your bootstrap connection may go through a published port, but the coordinator `Node` carries the *internal* advertised name. The client tries to connect, DNS/route fails, `NetworkClient` reports the node disconnected, and `AbstractCoordinator.coordinatorDead()` fires. Note the synthetic node id: the coordinator `Node` id is `Integer.MAX_VALUE - partition`. `2147483644` \= `2147483647 - 3`, i.e. your group hashes to `__consumer_offsets-3`. That giant id is the fingerprint of a coordinator node (versus a normal broker id like `1`).

**(B) `__consumer_offsets` has no leader / is loading.** The coordinator can only serve a group if the backing partition has an elected leader and its state is loaded into memory. On a single broker with the default `offsets.topic.replication.factor=3`, the internal topic can't be created (only one broker) — `FindCoordinator` returns `COORDINATOR_NOT_AVAILABLE` (error code 15). Right after a broker start or leadership change, the partition exists but is replaying its log — `COORDINATOR_LOAD_IN_PROGRESS` (code 14). Both are retriable, both surface as "unavailable or invalid," both self-heal *if* the topic can eventually get a leader. If RF is misconfigured for the cluster size, it never can, and you loop forever.

**(C) The coordinator broker actually went away.** In a real multi-broker cluster, if the broker leading your offsets partition dies or is rolling-restarted, in-flight heartbeats/commits get `NOT_COORDINATOR` (code 16) or a raw disconnect. The client correctly marks it dead and rediscovers — and once a new leader is elected for that offsets partition, rediscovery returns the new coordinator and the group heals in seconds. Transient (A)/(B) bursts during restarts are normal; a *persistent* loop is always a config or reachability problem, not bad luck.

The key insight: bootstrap reachability and coordinator reachability are **two independent connection paths**. The bootstrap servers are for metadata. The coordinator connection uses whatever the owning broker advertises. Any environment that makes those two paths resolve differently — port-publishing, split-horizon DNS, single-advertised-listener — breaks the second path while the first looks perfectly healthy.

## 5\. The Fix

**Fix for (A) — advertise an address the client can actually reach.** Use two listeners: an internal one for inter-broker/in-cluster traffic and an external one advertised with a host-reachable name.

```diff
  environment:
    KAFKA_NODE_ID: 1
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
-   KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
-   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+   KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
+   KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
-   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
+   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
-   KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+   KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

```

Now a host client bootstrapping on `localhost:9092` (EXTERNAL) gets a coordinator advertised as `localhost:9092` — reachable. A client running *inside* the Docker network bootstraps on `kafka:29092` (INTERNAL) and gets a coordinator advertised as `kafka:29092` — also reachable. The rule: **every client must bootstrap on the same listener whose advertised address it can reach**, because the coordinator will be handed back on that listener.

**Fix for (B) — make the internal topics able to elect a leader.** On a single-broker dev cluster, shrink the internal replication factors so `__consumer_offsets` can be created:

```properties
# single-broker dev only — never in prod
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

```

In production this class of "unavailable/loading" is transient by design — don't paper over it by lowering RF. Keep `offsets.topic.replication.factor=3`, ensure ≥3 brokers, and let rediscovery ride out the load window.

**Fix for (C) — nothing to fix in the client; make it patient.** The client already recovers. Make sure a heartbeat can survive a leader election without evicting you, and that request retries outlast a rolling restart:

```java
p.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 45000);      // default since 3.0
p.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 3000);    // ~1/3 of session
p.put(ConsumerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, 10000); // backoff, don't hot-loop

```

## 6\. Best Practices & The Better Design

The whole class of "coordinator dead" loops disappears when you treat listeners as a **first-class network contract**, not an afterthought. The right-way Compose block, verified:

```yaml
# Correct dual-listener KRaft broker
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"      # only the EXTERNAL listener is published to the host
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0   # faster group formation in dev

```

Design rules that make this a non-issue:

- **Each caller bootstraps on the listener it can reach.** Apps inside the cluster use the internal advertised name; laptops and CI use the external one. Never share a single `advertised.listeners` across both worlds.
- **In Kubernetes**, advertise the stable per-broker address clients use — the headless-service pod DNS name for in-cluster clients, or the per-broker NodePort/LoadBalancer/broker-specific Ingress host for external ones. A single shared Service VIP cannot work as an advertised listener, because the coordinator must be addressable as a *specific* broker.
- **Provision internal topics for the real cluster size.** RF=3 with ≥3 brokers in prod; only shrink RF in throwaway dev.
- **Kafka 4.0 / KIP-848** (new consumer group protocol, `group.protocol=consumer`) moves assignment to the broker but still routes group traffic to the coordinator on its advertised address — the listener contract above is unchanged and just as load-bearing.

## 7\. How to Prevent It Long-Term

- **Probe the coordinator path in CI, not just bootstrap.** A green `AdminClient.describeCluster()` only proves bootstrap works. Add a Testcontainers test that actually *subscribes and polls a record* through a consumer group — that's the only check that exercises the coordinator connection end-to-end.
- **Alert on rebalance/coordinator churn.** Watch the client metric `last-rebalance-seconds-ago` staying near zero and `assigned-partitions == 0` on a subscribed consumer — that combination is the "never joins" fingerprint. On brokers, alert on `OfflinePartitionsCount > 0` (an offline `__consumer_offsets` partition = a coordinator no one can reach).
- **Lint your compose/Helm listeners.** A CI check that fails when `advertised.listeners` contains an internal-only hostname on the external listener catches the #1 cause before it ships.
- **Verify with the tooling.** `kafka-consumer-groups.sh --describe --group <g> --state` prints the coordinator host:port — put it in your runbook so on-call can confirm reachability from the app's network namespace in seconds.

## 8\. Key Takeaways

- **"Marking the coordinator dead" / "coordinator is unavailable or invalid" = the group-coordinator connection failed**, a different path from bootstrap. It's INFO-level, so it silently loops instead of crashing.
- The synthetic node id `2147483644` \= `Integer.MAX_VALUE − partition`; it identifies a *coordinator* node and tells you which `__consumer_offsets` partition owns your group.
- **Bootstrap working proves nothing about coordinator reachability.** In Docker/K8s, the coordinator is handed back on the owning broker's advertised address — fix `advertised.listeners`, not the client.
- Single-broker dev needs `offsets.topic.replication.factor=1` or `__consumer_offsets` can't get a leader and every group loops on `COORDINATOR_NOT_AVAILABLE`.
- Transient bursts during broker restarts are normal and self-heal; a *persistent* loop is always config or networking — never wait it out.