Kafka ListenerNotFoundException: No Matching Listener Fix

You have a multi-broker cluster. The bootstrap connection succeeds. Metadata comes back. And then produce or consume hangs on some partitions while others work fine — or a broker log spits out ListenerNotFoundException and the client logs partitions have leader brokers without a matching listener. This is not a reachability problem in the usual "wrong host/port" sense. Every address in play is resolvable. The problem is that a listener name the client is routing on does not exist on the leader broker for a given partition.

This is the one advertised-listener failure that only bites in a multi-broker topology, and only for a subset of partitions — which is exactly why it survives single-broker local tests and detonates the moment you scale the cluster.

1. The Error

Broker side (server.log on the broker that received the metadata request):

org.apache.kafka.common.errors.ListenerNotFoundException: There is no listener on the leader broker that matches the listener on which metadata request was processed.

Java client side (kafka-clients Metadata / NetworkClient), logged at WARN and repeating every metadata.max.age.ms:

[Producer clientId=producer-1] 1 partitions have leader brokers without a matching listener, including [orders-2]

And the symptom that actually pages you — the produce future never completes, then:

org.apache.kafka.common.errors.TimeoutException: Topic orders not present in metadata after 60000 ms.

or, for a producer that got partial metadata:

org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for orders-2:120000 ms has passed since batch creation

ListenerNotFoundException is error code 72 (LISTENER_NOT_FOUND, added in KAFKA-6546). It extends InvalidMetadataException → RetriableException, so it is retriable: the client discards its metadata and refetches. That inheritance is the whole story — it is why a transient version (mid-rolling-restart) self-heals in seconds, and why a config version loops forever without ever throwing a fatal exception. It just silently never makes progress on the affected partitions.

Applies to: Apache Kafka 1.0 through 4.0, KRaft and ZooKeeper, kafka-clients 1.0+. Almost always seen on Docker Compose or Kubernetes multi-broker clusters, or managed setups where brokers were provisioned with asymmetric listener configs.

2. How to Reproduce It (step-by-step)

The trigger: a cluster where not every broker exposes the listener the client connects on. The cleanest repro is a two-broker KRaft cluster where broker 2 is missing the EXTERNAL listener that the host client bootstraps against.

docker-compose.yml (Apache Kafka 3.9.0, KRaft combined mode — broker 2 intentionally broken):

services:
  broker1:
    image: apache/kafka:3.9.0
    hostname: broker1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker1:9093,2@broker2:9093
      KAFKA_LISTENERS: INTERNAL://:29092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker1:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      CLUSTER_ID: 5L6g3nShT-eMCtK--X86sw

  broker2:
    image: apache/kafka:3.9.0
    hostname: broker2
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker1:9093,2@broker2:9093
      # BROKEN: no EXTERNAL listener at all — only INTERNAL + CONTROLLER
      KAFKA_LISTENERS: INTERNAL://:29092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker2:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      CLUSTER_ID: 5L6g3nShT-eMCtK--X86sw

Bring it up and create a topic with enough partitions that at least one lands on broker 2 as leader:

docker compose up -d

# create from inside broker1 (uses INTERNAL, which both brokers have)
docker compose exec broker1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server broker1:29092 \
  --create --topic orders --partitions 6 --replication-factor 2

# see which broker leads each partition
docker compose exec broker1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server broker1:29092 \
  --describe --topic orders

You will see some partitions with Leader: 2. Now produce from the host, which can only bootstrap on localhost:9092 (broker 1's EXTERNAL):

# from your host machine, with kafka-clients or the CLI pointed at localhost:9092
echo "test" | kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders

Messages routed to partitions led by broker 1 succeed. Messages routed to partitions led by broker 2 stall, and the client logs partitions have leader brokers without a matching listener, including [orders-N]. Broker 1's log (it processed the metadata request on EXTERNAL) records the ListenerNotFoundException while trying to resolve broker 2's EXTERNAL endpoint — which does not exist.

3. Why It Happens — Surface Level

A Kafka listener has three moving parts: a name (INTERNAL, EXTERNAL, PLAINTEXT…), a bind socket (listeners), and an advertised address (advertised.listeners). Clients do not connect "to a broker"; they connect on a listener. When you bootstrap on localhost:9092, you are speaking on the EXTERNAL listener, and Kafka promises to hand you back EXTERNAL advertised addresses for every broker that leads a partition you touch.

That promise breaks when a leader broker has no EXTERNAL listener. The broker serving your metadata request cannot fill in an EXTERNAL endpoint for that leader, so it either flags the partition with LISTENER_NOT_FOUND or the client sees a leader with no matching listener and skips it. Either way, that partition is invisible to you on that listener.

4. Why It Happens — Under the Hood

When a client sends a MetadataRequest, it arrives on a specific listener. The broker builds the MetadataResponse by walking every partition's leader and looking up that leader's advertised endpoint for the same listener name the request came in on. This is deliberate: a client that connected on EXTERNAL must be given EXTERNAL addresses for all brokers, because an INTERNAL address like broker2:29092 is meaningless from outside the Docker network.

The lookup lives in the broker's metadata cache. Each broker registers its advertised.listeners set — a map of listener name → host:port — into cluster metadata (the __cluster_metadata log in KRaft, ZooKeeper /brokers/ids in the legacy path). When broker 1 resolves broker 2's endpoint for listener EXTERNAL and finds no entry, there is nothing valid to return. Pre-KAFKA-6546 the broker returned a bogus/empty node; after it, the broker returns LISTENER_NOT_FOUND (code 72) so the client can distinguish "leader unknown" from "leader known but unreachable on your listener."

Because the exception is a subclass of InvalidMetadataException, the client treats it as a stale-metadata signal: drop the cache, refetch, retry. If listeners are genuinely mid-update (a rolling restart adding a listener), the refetch soon sees the new endpoint and the partition heals — this is the benign, transient case the javadoc calls out. If the config is simply asymmetric, every refetch returns the same result. The client loops, the partition never becomes writable, and the only fatal error you ever see is a downstream TimeoutException after max.block.ms / delivery.timeout.ms expire.

The same mechanism produces the client-side WARN partitions have leader brokers without a matching listener. Here the Java client itself, parsing a MetadataResponse, finds a partition whose leader node id has no address for the listener the client is using, and warns rather than throwing. Two faces, one root cause: the listener you're on doesn't exist on the leader.

5. The Fix

Make every broker expose every listener that any client class needs. In the repro, broker 2 must gain the EXTERNAL listener with a host-unique advertised port.

Before (broker 2, broken):

      KAFKA_LISTENERS: INTERNAL://:29092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker2:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT

After (broker 2, fixed — note the distinct host port 9094):

    ports:
      - "9094:9094"
    environment:
      KAFKA_LISTENERS: INTERNAL://:29092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker2:29092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT

The rule that must hold across the whole cluster: the set of listener names is identical on every broker, and each broker's EXTERNAL advertised address is individually reachable from the client's network. Two brokers cannot both advertise EXTERNAL://localhost:9092; each needs its own host port (9092, 9094, …) mapped through.

Verify the fix by asking each broker what it advertises, on the listener you care about:

# from the host, on the EXTERNAL listener — must succeed for BOTH brokers' ports
kafka-broker-api-versions.sh --bootstrap-server localhost:9092 | grep -i "id:"
kafka-broker-api-versions.sh --bootstrap-server localhost:9094 | grep -i "id:"

Choosing the fix by environment:

• Rolling change added a listener to some brokers only — this is the transient case; finish the rollout so every broker has the listener, then the retriable error clears itself. Don't tune timeouts; just complete the change.
• Docker Compose / local — give each broker a unique EXTERNAL host port as above.
• Kubernetes — do not advertise a shared Service VIP for the external listener; each broker needs a stable, individually addressable endpoint (per-broker NodePort/LoadBalancer or per-pod DNS via a headless Service). A single VIP means metadata hands back one address for every leader and clients land on the wrong broker.

6. Best Practices & The Better Design

Treat listeners as a cluster-wide network contract, not per-broker copy-paste. The failure exists only because one broker's listener set diverged from the others'. The design that removes the whole class of problem:

• One canonical listener template applied to all brokers, parameterized only by node id and host port. Every broker defines the same listener names — INTERNAL for inter-broker traffic, EXTERNAL for off-cluster clients, CONTROLLER for the KRaft quorum. No broker is allowed to omit a client-facing listener.
• inter.broker.listener.name on its own dedicated listener (INTERNAL), never shared with external clients. This keeps replication traffic on an address that is always reachable broker-to-broker regardless of what external clients do.
• Per-broker external addressing. Each broker advertises its own external host:port. The invariant to encode in review: for every listener name L and every broker B, B advertises an L endpoint, and that endpoint is reachable from every client that bootstraps on L.

A correct multi-broker KRaft broker config, generated from a template:

# broker N (node.id=N, unique EXTERNAL host port assigned per broker)
process.roles=broker,controller
node.id=${NODE_ID}
controller.quorum.voters=1@broker1:9093,2@broker2:9093,3@broker3:9093
listeners=INTERNAL://:29092,CONTROLLER://:9093,EXTERNAL://:${EXTERNAL_PORT}
advertised.listeners=INTERNAL://${HOST}:29092,EXTERNAL://${EXTERNAL_HOST}:${EXTERNAL_PORT}
listener.security.protocol.map=INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
inter.broker.listener.name=INTERNAL
controller.listener.names=CONTROLLER

The key discipline: the only per-broker variables are NODE_ID, HOST, EXTERNAL_HOST, and EXTERNAL_PORT. The listener names never vary. When names are templated and only addresses differ, a broker can never silently lack a listener another broker has.

7. How to Prevent It Long-Term

• Config-as-code with a linter. Render broker configs from one template (Strimzi Kafka CRD, Helm, Terraform) and add a CI check that asserts the listener name set is identical across all brokers and that no two brokers advertise the same external host:port. This catches the divergence before deploy.
• Post-deploy reachability probe. For each broker's external endpoint, run kafka-broker-api-versions.sh --bootstrap-server <that endpoint> from the client network namespace (a separate container/pod, not exec inside a broker where internal DNS resolves for free). A green bootstrap on one broker proves nothing about the others; probe them all.
• Alerting. There is no single broker metric named "clients can't route to me," so watch the fingerprints: client-side ListenerNotFoundException / "matching listener" WARNs, producers with rising record-error-rate on specific partitions, and new consumers stuck at assigned-partitions=0 with climbing lag. Alert on the client logs, since the broker side is only a retriable metadata response.
• Test at real broker count. A single-broker Testcontainers test cannot reproduce this — the leader is always the broker you're connected to. Add an integration test with at least three brokers and a topic whose partitions spread across all of them, produced to and consumed from the external listener. That test fails loudly the instant one broker's listener config drifts.
• Never add a listener to part of the cluster and stop. If you're introducing a new listener (say, an SSL external one), roll it to every broker in the same change. A half-applied listener rollout is the benign-but-noisy version of this error; leaving it half-applied makes it permanent.

8. Key Takeaways / Learnings

• ListenerNotFoundException (code 72, retriable) means the leader broker has no endpoint on the listener your client connected on — it is a listener-name mismatch, not a wrong host/port.
• It only appears in multi-broker clusters and only for partitions led by the misconfigured broker, so it hides in single-broker local tests and surfaces on scale-out.
• Because it's a RetriableException, the client loops silently and you only see a downstream TimeoutException; the real diagnosis is in the WARN partitions have leader brokers without a matching listener.
• The fix and the prevention are the same principle: every broker exposes the same listener names, with per-broker reachable addresses — never a shared VIP, never an omitted listener.
• Template listener configs so only addresses vary, lint that the name sets match across brokers, and probe every broker's external endpoint from the client's network — not from inside the cluster.