Kafka: Marking the Coordinator Dead

Reading Progress

Discover → dead → discover → dead. The consumer never joins the group, never gets an assignment, poll() returns empty forever, and lag climbs. This is one of the most-pasted lines in Kafka support history, and 90% of the time in a POC it is not a broker problem — it's a networking problem hiding behind a friendly-looking INFO log.

1. The Error

The message originates in AbstractCoordinator (the base class behind ConsumerCoordinator) in kafka-clients. It is logged at INFO, which is why people miss it — nothing throws, nothing crashes, the app just does nothing.

Client: org.apache.kafka:kafka-clients 3.x (wording confirmed on 3.6–3.9; identical mechanics on 4.0). Spring Kafka 3.x wraps the same client.
Broker: Kafka 3.x / 4.0, KRaft mode.
Where it bites: Docker Compose and Kubernetes POCs, and any multi-listener cluster where the client can reach the bootstrap endpoint but not the coordinator's advertised endpoint. Also real production incidents when the coordinator broker dies or __consumer_offsets has no leader.

The tell that separates this from a plain bootstrap failure (Connection to node -1 could not be established): here the client already has cluster metadata and has resolved a specific coordinator node with a synthetic id like 2147483644. It is one hop further along than a bootstrap failure.

2. How to Reproduce It

The cleanest reproduction is the Docker listener trap: advertise a hostname the host machine can't resolve. Bootstrap discovery works because you connect by IP/port, but the coordinator gets advertised under an internal name.

# docker-compose.yml — Kafka 3.9 KRaft, deliberately broken listeners
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      # BUG: advertises the internal container name to host clients
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      # single broker: shrink internal topic RF so it can actually get a leader
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

Consumer, run from the host (not inside a container):

// kafka-clients 3.9.0
Properties p = new Properties();
p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");
p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
p.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

try (KafkaConsumer<String, String> c = new KafkaConsumer<>(p)) {
    c.subscribe(List.of("orders"));
    while (true) {
        var records = c.poll(Duration.ofSeconds(1));
        System.out.println("got " + records.count());
    }
}

localhost:9092 works for bootstrap — the port is published — so metadata loads. But the coordinator comes back advertised as kafka:29092 (or kafka:9092), which does not resolve from your laptop. The consumer discovers the coordinator, fails to open a connection to it, marks it dead, and loops. Turn on log4j at INFO for org.apache.kafka.clients.consumer.internals and you'll see the discover/dead cycle immediately.

To confirm which __consumer_offsets partition (and therefore which broker) owns your group:

docker exec -it kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group order-service --state

The COORDINATOR column shows the host:port the group is pinned to. If that host:port is unreachable from where your client runs, you've found it.

3. Why It Happens — Surface Level

A consumer group is managed by exactly one broker: the group coordinator. To find it, the client sends a FindCoordinator request and gets back the broker that leads the __consumer_offsets partition owning your group. The client then opens a separate connection to that broker — using the address the broker advertises — for all group traffic: JoinGroup, SyncGroup, Heartbeat, OffsetCommit, OffsetFetch.

"Marking the coordinator dead" means that separate coordinator connection failed, or the coordinator returned a "not me / not ready" error. The client drops its cached coordinator and re-runs FindCoordinator. If the underlying cause is stable (wrong advertised address, missing internal topic, dead broker), rediscovery returns the same bad answer and you loop.

4. Why It Happens — Under the Hood

Three mechanically distinct failure modes produce the identical log line.

(A) Coordinator advertised on an address the client can't reach. This is the Docker/K8s classic. FindCoordinator doesn't return an abstract "broker 1" — it returns a Node built from that broker's advertised.listeners for the listener the client is using. Your bootstrap connection may go through a published port, but the coordinator Node carries the internal advertised name. The client tries to connect, DNS/route fails, NetworkClient reports the node disconnected, and AbstractCoordinator.coordinatorDead() fires. Note the synthetic node id: the coordinator Node id is Integer.MAX_VALUE - partition. 2147483644 = 2147483647 - 3, i.e. your group hashes to __consumer_offsets-3. That giant id is the fingerprint of a coordinator node (versus a normal broker id like 1).

(B) __consumer_offsets has no leader / is loading. The coordinator can only serve a group if the backing partition has an elected leader and its state is loaded into memory. On a single broker with the default offsets.topic.replication.factor=3, the internal topic can't be created (only one broker) — FindCoordinator returns COORDINATOR_NOT_AVAILABLE (error code 15). Right after a broker start or leadership change, the partition exists but is replaying its log — COORDINATOR_LOAD_IN_PROGRESS (code 14). Both are retriable, both surface as "unavailable or invalid," both self-heal if the topic can eventually get a leader. If RF is misconfigured for the cluster size, it never can, and you loop forever.

(C) The coordinator broker actually went away. In a real multi-broker cluster, if the broker leading your offsets partition dies or is rolling-restarted, in-flight heartbeats/commits get NOT_COORDINATOR (code 16) or a raw disconnect. The client correctly marks it dead and rediscovers — and once a new leader is elected for that offsets partition, rediscovery returns the new coordinator and the group heals in seconds. Transient (A)/(B) bursts during restarts are normal; a persistent loop is always a config or reachability problem, not bad luck.

The key insight: bootstrap reachability and coordinator reachability are two independent connection paths. The bootstrap servers are for metadata. The coordinator connection uses whatever the owning broker advertises. Any environment that makes those two paths resolve differently — port-publishing, split-horizon DNS, single-advertised-listener — breaks the second path while the first looks perfectly healthy.

5. The Fix

Fix for (A) — advertise an address the client can actually reach. Use two listeners: an internal one for inter-broker/in-cluster traffic and an external one advertised with a host-reachable name.

  environment:
    KAFKA_NODE_ID: 1
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
-   KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
-   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+   KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
+   KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
-   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
+   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
-   KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+   KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

Now a host client bootstrapping on localhost:9092 (EXTERNAL) gets a coordinator advertised as localhost:9092 — reachable. A client running inside the Docker network bootstraps on kafka:29092 (INTERNAL) and gets a coordinator advertised as kafka:29092 — also reachable. The rule: every client must bootstrap on the same listener whose advertised address it can reach, because the coordinator will be handed back on that listener.

Fix for (B) — make the internal topics able to elect a leader. On a single-broker dev cluster, shrink the internal replication factors so __consumer_offsets can be created:

# single-broker dev only — never in prod
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

In production this class of "unavailable/loading" is transient by design — don't paper over it by lowering RF. Keep offsets.topic.replication.factor=3, ensure ≥3 brokers, and let rediscovery ride out the load window.

Fix for (C) — nothing to fix in the client; make it patient. The client already recovers. Make sure a heartbeat can survive a leader election without evicting you, and that request retries outlast a rolling restart:

p.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 45000);      // default since 3.0
p.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 3000);    // ~1/3 of session
p.put(ConsumerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, 10000); // backoff, don't hot-loop

6. Best Practices & The Better Design

The whole class of "coordinator dead" loops disappears when you treat listeners as a first-class network contract, not an afterthought. The right-way Compose block, verified:

# Correct dual-listener KRaft broker
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"      # only the EXTERNAL listener is published to the host
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0   # faster group formation in dev

Design rules that make this a non-issue:

Each caller bootstraps on the listener it can reach. Apps inside the cluster use the internal advertised name; laptops and CI use the external one. Never share a single advertised.listeners across both worlds.
In Kubernetes, advertise the stable per-broker address clients use — the headless-service pod DNS name for in-cluster clients, or the per-broker NodePort/LoadBalancer/broker-specific Ingress host for external ones. A single shared Service VIP cannot work as an advertised listener, because the coordinator must be addressable as a specific broker.
Provision internal topics for the real cluster size. RF=3 with ≥3 brokers in prod; only shrink RF in throwaway dev.
Kafka 4.0 / KIP-848 (new consumer group protocol, group.protocol=consumer) moves assignment to the broker but still routes group traffic to the coordinator on its advertised address — the listener contract above is unchanged and just as load-bearing.

7. How to Prevent It Long-Term

Probe the coordinator path in CI, not just bootstrap. A green AdminClient.describeCluster() only proves bootstrap works. Add a Testcontainers test that actually subscribes and polls a record through a consumer group — that's the only check that exercises the coordinator connection end-to-end.
Alert on rebalance/coordinator churn. Watch the client metric last-rebalance-seconds-ago staying near zero and assigned-partitions == 0 on a subscribed consumer — that combination is the "never joins" fingerprint. On brokers, alert on OfflinePartitionsCount > 0 (an offline __consumer_offsets partition = a coordinator no one can reach).
Lint your compose/Helm listeners. A CI check that fails when advertised.listeners contains an internal-only hostname on the external listener catches the #1 cause before it ships.
Verify with the tooling. kafka-consumer-groups.sh --describe --group <g> --state prints the coordinator host:port — put it in your runbook so on-call can confirm reachability from the app's network namespace in seconds.

8. Key Takeaways

"Marking the coordinator dead" / "coordinator is unavailable or invalid" = the group-coordinator connection failed, a different path from bootstrap. It's INFO-level, so it silently loops instead of crashing.
The synthetic node id 2147483644 = Integer.MAX_VALUE − partition; it identifies a coordinator node and tells you which __consumer_offsets partition owns your group.
Bootstrap working proves nothing about coordinator reachability. In Docker/K8s, the coordinator is handed back on the owning broker's advertised address — fix advertised.listeners, not the client.
Single-broker dev needs offsets.topic.replication.factor=1 or __consumer_offsets can't get a leader and every group loops on COORDINATOR_NOT_AVAILABLE.
Transient bursts during broker restarts are normal and self-heal; a persistent loop is always config or networking — never wait it out.

apache-kafka kafka-errors kafka-consumer consumer-groups group-coordinator kafka-docker advertised-listeners Java

Gopi Gorantala Twitter

I'm Gopi — 15+ years in Java, building Kafka and Flink platforms for banks, where one lost event is a financial discrepancy. I write javahandbook.com because the guides I needed didn't exist. Everything here is tested against a real cluster first.

On This Page