Kafka NotLeaderOrFollowerException: Cause and Fix

Your producer log fills with warnings like this:

[Producer clientId=producer-1] Received invalid metadata error in produce request
on partition orders-2 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException:
For requests intended only for the leader, this error indicates that the broker is not
the current leader. For requests intended for any replica, this error indicates that
the broker is not a replica of the topic partition.. Going to request metadata update now

Or the protocol-level variant during metadata fetches and console tooling:

WARN [Producer clientId=console-producer] Error while fetching metadata with
correlation id 5 : {orders=NOT_LEADER_OR_FOLLOWER}

On the consumer side:

[Consumer clientId=consumer-orders-1, groupId=orders] Received unknown topic or
partition error in fetch for partition orders-2
org.apache.kafka.common.errors.NotLeaderOrFollowerException

If you're on kafka-clients < 2.6 or an older broker, the same error code (6) is named differently — it's the identical failure:

org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not
the leader for that topic-partition.

NOT_LEADER_FOR_PARTITION was renamed to NOT_LEADER_OR_FOLLOWER in Kafka 2.6 alongside KIP-392 (fetch from followers), because fetch requests may now legitimately target followers. Same error code on the wire, so a 3.x client talking to an old broker still maps it correctly.

Typical setup: kafka-clients 3.x/4.x, Spring Boot 3.x, either a multi-broker cluster during rolling restarts/reassignment, or a multi-broker Docker Compose stack with misconfigured advertised listeners. Verified against kafka-clients 3.9.0 and apache/kafka 3.9 KRaft brokers.

There are two very different situations hiding behind this one exception, and the fix depends entirely on which you have:

1. Transient: it appears in bursts during broker restarts or leader elections, then stops. This is Kafka working as designed — but it becomes a real failure if your producer config turns a retriable blip into a TimeoutException.
2. Persistent: it loops forever. The client never converges on the right leader. This is almost always a listener/port misconfiguration, and no amount of retrying fixes it.

2. How to Reproduce It (step-by-step)

Three-broker KRaft cluster:

# docker-compose.yml
version: "3.8"
x-kafka-common: &kafka-common
  image: apache/kafka:3.9.0
  environment: &kafka-env
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    CLUSTER_ID: 5L6g3nShT-eMCtK--X86sw

services:
  kafka1:
    <<: *kafka-common
    ports: ["19092:19092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 1
      KAFKA_LISTENERS: CONTROLLER://:9093,INTERNAL://:9092,EXTERNAL://:19092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL://localhost:19092
  kafka2:
    <<: *kafka-common
    ports: ["29092:29092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 2
      KAFKA_LISTENERS: CONTROLLER://:9093,INTERNAL://:9092,EXTERNAL://:29092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:9092,EXTERNAL://localhost:29092
  kafka3:
    <<: *kafka-common
    ports: ["39092:39092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 3
      KAFKA_LISTENERS: CONTROLLER://:9093,INTERNAL://:9092,EXTERNAL://:39092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:9092,EXTERNAL://localhost:39092

docker compose up -d

docker compose exec kafka1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 \
  --create --topic orders --partitions 6 --replication-factor 3

Trigger the transient variant. Start a producer, then kill a leader mid-stream:

# terminal 1: steady producer traffic
docker compose exec kafka1 /opt/kafka/bin/kafka-verifiable-producer.sh \
  --bootstrap-server kafka1:9092 --topic orders --throughput 500

# terminal 2: find which broker leads partition 0, then kill it
docker compose exec kafka1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 --describe --topic orders
docker compose stop kafka2   # assuming kafka2 led some partitions

The instant kafka2 dies, every in-flight produce request routed to it fails. The controller elects new leaders from the ISR within a few hundred milliseconds, but the producer's metadata still says "kafka2 leads orders-2" — the next produce to the new connection target answers with NOT_LEADER_OR_FOLLOWER. You'll see the warning burst, a metadata refresh, then silence. Restart kafka2 and wait ~5 minutes and you may see a second, smaller burst — that's auto.leader.rebalance.enable=true moving leadership back to the preferred replica (more below).

You can also trigger it without killing anything:

docker compose exec kafka1 /opt/kafka/bin/kafka-leader-election.sh \
  --bootstrap-server kafka1:9092 --election-type PREFERRED --all-topic-partitions

Trigger the persistent variant. Change all three brokers to advertise the same external endpoint — a classic copy-paste error in Compose files:

# WRONG — all three brokers advertise localhost:19092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:9092,EXTERNAL://localhost:19092

Now produce from the host to a partition whose leader is broker 2 or 3. The metadata response says "leader for orders-2 is at localhost:19092" — but that port is mapped to broker 1. The client dutifully sends the produce request to broker 1, broker 1 answers NOT_LEADER_OR_FOLLOWER, the client refreshes metadata, gets the same wrong answer, and loops until delivery.timeout.ms expires:

org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for
orders-2:120000 ms has passed since batch creation

That final TimeoutException is what most people see and search for — the NOT_LEADER_OR_FOLLOWER warnings scrolling past above it are the actual story.

3. Why It Happens — Surface Level

Kafka clients don't ask a load balancer where to send data. Every client caches a metadata map — for each partition: leader broker, its host/port (from advertised.listeners), and the replica set. Produce requests go directly to the cached leader; fetch requests go to the leader (or a follower, with KIP-392 rack-aware fetching).

NotLeaderOrFollowerException means exactly one thing: the broker that received the request is not the leader (or a valid replica) for that partition anymore — the client's metadata is stale. Leadership moved because a broker restarted, the controller failed over, a reassignment completed, or preferred leader election ran. The error is the broker telling the client "wrong address, go re-ask."

It's classified as a retriable InvalidMetadataException subclass — the client automatically requests a metadata update and retries. That's why the transient variant is a WARN, not an ERROR.

4. Why It Happens — Under the Hood

The election path. In KRaft, the active controller detects broker failure via missed broker heartbeats (broker.session.timeout.ms, default 9s) — or immediately, on a controlled shutdown request. It picks the new leader as the first ISR member in the replica list, bumps the partition's leader epoch, and writes the change to the __cluster_metadata log, which all brokers replay. Sub-second for a handful of partitions; on a broker leading thousands of partitions, the metadata propagation and log recovery dominate failover time.

Why the client keeps hitting the old leader. The producer refreshes metadata on demand (when an InvalidMetadataException comes back) and periodically (metadata.max.age.ms, default 5 minutes). Between the election and the refresh, every produce to a moved partition fails with NOT_LEADER_OR_FOLLOWER. The client also validates leader epochs (KIP-320): if a metadata response carries an older epoch than what the client already saw, it's discarded — protecting you from a stale broker's metadata cache re-poisoning the client during churn.

Why it comes back 5 minutes after a restart. Each partition has a preferred leader — the first replica in its assignment. After you restart a broker, it rejoins as a follower everywhere. With auto.leader.rebalance.enable=true (default), the controller checks every leader.imbalance.check.interval.seconds (300) whether more than leader.imbalance.per.broker.percentage (10%) of leadership sits off-preferred, and if so moves it back — another round of elections, another burst of NOT_LEADER_OR_FOLLOWER warnings. This is the "we restarted the broker, everything was fine, then 5 minutes later we got errors again" mystery.

Why the persistent variant never converges. The retry loop assumes the next metadata refresh yields a routable leader address. In the broken-Compose case, metadata is correct (leader is broker 2) but the advertised address for broker 2 points at broker 1's mapped port. Stale metadata isn't the problem — wrong metadata is — so metadata refreshes can't help. Same failure shape appears in Kubernetes when a NodePort/LoadBalancer service fronts all brokers with one address, or when a proxy terminates connections without per-broker routing (this is exactly why per-broker routes exist in Strimzi and MSK's per-broker endpoints).

Where it hurts even with retries. Retries protect completed send() calls, but each retried batch still burns time against delivery.timeout.ms (120s default) — a slow failover on a busy partition can expire batches. And with retries effectively unbounded but max.in.flight.requests.per.connection > 1 without idempotence, retried batches can land out of order after the new leader takes over.

5. The Fix

Fix 1: Transient bursts — make retries actually cover the failover window

This is the default-config path and mostly means not breaking the defaults:

 # producer.properties
-acks=1
-retries=0
-delivery.timeout.ms=15000
+acks=all
+enable.idempotence=true          # implies retries=MAX_INT, safe reordering guard
+delivery.timeout.ms=120000       # must cover: leader failover + linger.ms + request timeouts
+metadata.max.age.ms=300000       # default is fine; on-demand refresh handles elections

With enable.idempotence=true (default since 3.0 with acks=all), a NOT_LEADER_OR_FOLLOWER burst during a rolling restart is invisible to your application: batches retry against the new leader with sequence numbers intact, no duplicates, no reordering. If you had retries=0, each blip surfaced as a failed send — that's self-inflicted.

Fix 2: Persistent loop in Docker/K8s — one advertised endpoint per broker

Every broker must advertise an address that routes to that broker specifically from the client's network:

 # kafka2 environment
-KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:9092,EXTERNAL://localhost:19092
+KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:9092,EXTERNAL://localhost:29092

 # kafka2 ports
-    ports: ["19092:19092"]
+    ports: ["29092:29092"]

Verify what each broker actually advertises — from the client's side of the network:

docker run --rm --network host apache/kafka:3.9.0 \
  /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:19092 \
  | grep -E '^[a-z0-9.-]+:[0-9]+'
# must print THREE distinct host:port lines, one per broker

If that prints the same host:port three times, you've found your bug.

Fix 3: Restart-induced churn — controlled shutdown and rebalance tuning

For planned restarts, make sure you're getting controlled shutdown (default controlled.shutdown.enable=true; use docker compose stop, not kill -9) — the broker proactively hands leadership off before closing, shrinking the error window from seconds to near zero.

If the post-restart preferred-leader flapping bothers you, widen the tolerance rather than disabling the mechanism:

# broker config — fewer, larger rebalance passes
leader.imbalance.check.interval.seconds=900
leader.imbalance.per.broker.percentage=20

Setting auto.leader.rebalance.enable=false is valid on clusters where you run kafka-leader-election.sh --election-type PREFERRED from a maintenance job at a chosen time — but never disable it and then also never run elections, or one restarted broker ends up leading nothing while its peers overload.

6. Best Practices & The Better Design

The design goal: leader elections are routine in Kafka — rolling restarts, reassignments, cruise-control moves all cause them. Your clients and topics should absorb them silently.

// The producer that survives any leader election, spring-kafka 3.3.x
@Bean
public ProducerFactory<String, Order> producerFactory() {
    Map<String, Object> props = Map.of(
        ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:19092,localhost:29092,localhost:39092",
        ProducerConfig.ACKS_CONFIG, "all",
        ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true,
        ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000,
        ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 30_000,
        ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class,
        ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class
    );
    return new DefaultKafkaProducerFactory<>(props);
}

Pair it with topics that can actually elect a new leader when one broker is down: replication-factor=3, min.insync.replicas=2, and unclean.leader.election.enable=false (default). RF=1 topics turn every broker restart into partition unavailability — the NOT_LEADER_OR_FOLLOWER burst becomes an outage because there is no follower to elect. RF=2 with min.isr=2 blocks writes during every single-broker restart. RF=3/min.isr=2 is the floor for anything you care about.

List all bootstrap servers, not one. Bootstrap is only used for the first metadata fetch, but if the single bootstrap broker is the one being restarted, a newly started client can't fetch metadata at all — a different error, same incident.

And treat the log level as signal: kafka-clients logs NOT_LEADER_OR_FOLLOWER at WARN precisely because it's expected-and-recovered. Alert on send failures and record-error-rate, not on the warning text.

7. How to Prevent It Long-Term

Monitor election activity, not the client warning. On brokers, watch kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs for election frequency/latency and UncleanLeaderElectionsPerSec (must be zero — an unclean election means data loss, and clients see the same NOT_LEADER errors during it). A steady baseline election rate outside deploy windows means leadership flapping — usually a broker riding the edge of broker.session.timeout.ms from GC pauses or saturated request handlers.

Client-side, watch the producer metrics that failovers move: record-error-rate (real failures), record-retry-rate (elections happening), metadata-age. A retry-rate spike with zero error-rate during a deploy is a healthy cluster; error-rate following retry-rate means your delivery.timeout.ms isn't covering failover.

Chaos-test the failover path before production does it for you. A one-line test in staging: run kafka-verifiable-producer.sh and kafka-verifiable-consumer.sh, kill the partition leader, assert zero lost/duplicated acked messages. If that fails, your acks/idempotence/RF settings are wrong and no dashboard will save you.

CI-check the listener wiring. For every environment's Compose/Helm values, assert that the advertised external endpoints are pairwise distinct and each resolves to the right broker (the kafka-broker-api-versions.sh check above scripts cleanly). This catches the persistent variant at review time, where it's a 30-second fix instead of a production incident.

During planned maintenance, restart brokers one at a time, wait for URP (kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions) to return to zero between nodes, and let controlled shutdown migrate leadership. Reassignments via cruise-control or kafka-reassign-partitions.sh should be throttled so followers stay in the ISR — falling out of ISR mid-reassignment multiplies elections.

8. Key Takeaways

• NotLeaderOrFollowerException = the client's cached metadata points at a broker that no longer leads the partition. It's the renamed NotLeaderForPartitionException (Kafka 2.6+, KIP-392), error code 6, and it's retriable by design.
• Transient bursts during restarts/elections are normal. With acks=all + enable.idempotence=true + default delivery.timeout.ms, the client absorbs them with zero application impact. retries=0 turns routine elections into failed sends.
• A persistent loop means metadata refresh can't help — almost always advertised listeners routing multiple brokers to one endpoint (Docker port copy-paste, single K8s service for all brokers). Verify with kafka-broker-api-versions.sh: one distinct endpoint per broker, or it's broken.
• The mysterious "errors again 5 minutes after the restart" is auto.leader.rebalance.enable moving leadership back to preferred replicas. Tune the interval/threshold or schedule preferred elections — don't just disable it.
• Alert on record-error-rate, UncleanLeaderElectionsPerSec, and election rate baselines — not on the WARN text in client logs.