Kafka LEADER_NOT_AVAILABLE: Why It Happens and How to Fix It

You start a producer, aim it at a brand-new topic, and the console lights up:

WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 3 : {orders=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 4 : {orders=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 5 : {orders=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Two very different things hide behind that single line. Sometimes it clears in a second and your record lands — a transient election blip you should ignore. Sometimes it loops forever until max.block.ms fires and you get:

org.apache.kafka.common.errors.TimeoutException: Topic orders not present in metadata after 60000 ms.

This article separates the benign case from the broken one, and gives you the fix for each.

1. The Error

LEADER_NOT_AVAILABLE is Kafka error code 5 (LeaderNotAvailableException). The broker is telling the client: this topic-partition exists (or is being created), but right now there is no leader replica you can produce to or fetch from. It is a retriable error — the client is supposed to refresh metadata and try again, which is exactly why it surfaces as a WARN in NetworkClient and not a thrown exception, at least at first.

Do not confuse it with its two neighbors:

Versions where this shows up unchanged: kafka-clients 3.x–3.9 and 4.0, Spring Kafka 3.x, against KRaft or ZooKeeper brokers. The message wording (Error while fetching metadata with correlation id ...) has been stable since 2.x, so search hits from older posts still apply.

Typical setups where it bites:

• Docker/local POC — first produce to an auto-created or just-created topic.
• Managed Kafka / K8s — during a rolling broker restart or a partition reassignment.
• CI — a test produces immediately after AdminClient.createTopics() returns.

2. How to Reproduce It

Minimal KRaft cluster

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

docker compose up -d

Trigger A — transient (auto-creation race)

Produce to a topic that doesn't exist yet, with auto-create on:

docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:19092 \
  --topic orders

The first send races topic creation. The controller creates orders, but the leader for partition 0 isn't elected and propagated to the broker's metadata cache the instant the producer asks. You'll see one or two LEADER_NOT_AVAILABLE WARNs, then it clears and your message goes through. This is normal. Kafka's own docs and the kafka-topics.sh tooling emit this on first touch of a new topic.

Trigger B — persistent (broker restart mid-produce)

Start a steady producer, then bounce the broker while it runs:

// LeaderNotAvailableDemo.java  (kafka-clients 3.9.0)
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 60000);

try (var producer = new KafkaProducer<String, String>(p)) {
    for (int i = 0; ; i++) {
        producer.send(new ProducerRecord<>("orders", "k" + i, "v" + i),
            (md, ex) -> { if (ex != null) ex.printStackTrace(); });
        Thread.sleep(500);
    }
}

# while it runs, in another terminal:
docker restart kafka

During the restart window the partition has no online leader. The producer parks the batch and logs LEADER_NOT_AVAILABLE repeatedly until the broker comes back and elects a leader. On a single-broker cluster (RF=1) there is no other replica to take over, so the gap lasts the full restart. On RF≥3 with a healthy ISR, a follower is promoted in well under a second and you'd barely notice.

Trigger C — the one that never clears

Misconfigure advertised.listeners so the client can reach the bootstrap endpoint but the returned leader host is unroutable, or create a topic whose partitions can't get a leader (e.g. all replicas offline). The WARN loops until max.block.ms and you get the TimeoutException: Topic orders not present in metadata after 60000 ms. This is the case people actually need to fix.

3. Why It Happens — Surface Level

A Kafka producer can't send to a partition until it knows which broker leads it. It gets that from a MetadataResponse. If the partition currently has no leader — because it's mid-creation, mid-election, or its replicas are offline — the broker returns LEADER_NOT_AVAILABLE for that topic in the metadata response. The client caches nothing usable, waits retry.backoff.ms, and asks again.

So the WARN is not itself a failure. It's the client doing the right thing: retrying a retriable metadata condition. It only becomes a real problem when the condition never resolves, because then the retries exhaust max.block.ms (in send()/metadata wait) or delivery.timeout.ms (for a batch already accepted).

4. Why It Happens — Under the Hood

Every partition has a leader replica; all reads and writes go through it. The controller (the KRaft quorum controller, or the ZooKeeper-era controller broker) decides leadership and publishes it. When leadership is unknown, the metadata leader field is set to -1 and the partition carries the LEADER_NOT_AVAILABLE error.

There are three mechanically distinct ways to land there:

Topic creation lag. createTopics (or auto-creation) is asynchronous with respect to the produce path. The controller writes the topic to the metadata log, then leaders get assigned and the assignment propagates to each broker's MetadataCache. A producer that fetches metadata inside that window sees the topic with no leader yet. Under KRaft this window is small but non-zero; the broker applies metadata records from the controller as a log it replays, so there's inherent propagation delay.

Leader election in flight. During a controlled shutdown, rolling restart, or preferred-leader rebalance, the current leader steps down and a new one is elected from the ISR. Between step-down and the new leader's election being applied on the broker your client is talking to, that partition reports no leader. Cleanly-run clusters make this sub-second; the exposure scales with how fast the controller elects and how fast each broker applies the update.

No electable replica. If every replica for the partition is offline, or min.insync.replicas can't be met and unclean election is disabled (unclean.leader.election.enable=false, the safe default), there is nothing to elect. LEADER_NOT_AVAILABLE then persists until you restore a replica. This is the genuinely broken state, and it usually rides alongside under-replicated-partition alarms.

The client side is governed by two clocks. max.block.ms (default 60s) bounds how long send() blocks waiting for metadata before throwing TimeoutException: Topic ... not present in metadata. Once a batch is accepted into the accumulator, delivery.timeout.ms (default 120s) is the total budget for delivery including these metadata retries. retry.backoff.ms (default 100ms) paces the retries. So a "flood" of WARNs is just (elapsed / retry.backoff.ms) lines — noisy, but bounded.

5. The Fix

First decide which case you're in. If the WARN clears within a second or two and records land, there is nothing to fix — see best practices for silencing the noise. If it persists to a TimeoutException, fix the cluster or the config.

Fix A — don't produce before the topic has a leader (POC / CI)

Create the topic and wait for it, instead of racing auto-creation:

- // producer sends immediately, racing auto-creation
- producer.send(new ProducerRecord<>("orders", key, value));
+ try (Admin admin = Admin.create(Map.of(
+         AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"))) {
+     admin.createTopics(List.of(new NewTopic("orders", 3, (short) 1)))
+          .all().get();               // block until controller acks
+ }
+ // now the leader is elected before the first send
+ producer.send(new ProducerRecord<>("orders", key, value));

createTopics(...).all().get() returning does not guarantee the leader has propagated to every broker's cache, so in tight CI loops also give the producer a real budget rather than fighting the race:

  Properties p = new Properties();
- p.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 5000);
+ p.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 60000);   // absorb propagation lag

Fix B — turn off auto-creation so the race can't happen

Auto-created topics are the single biggest source of these WARNs in dev because the produce path is the creation trigger. Turn it off and provision topics explicitly:

# broker
auto.create.topics.enable=false

Now a missing topic fails fast and loudly (UNKNOWN_TOPIC_OR_PARTITION → TimeoutException) instead of half-creating under load. You create topics deliberately via AdminClient, kafka-topics.sh, or infra-as-code.

Fix C — persistent case: restore a leader

If it never clears, the partition has no electable leader. Check it:

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:19092 \
  --describe --topic orders

Topic: orders  Partition: 0  Leader: none  Replicas: 2  Isr:

Leader: none with an empty ISR means every replica is down. Bring the replica brokers back, or if a broker is permanently gone, reassign the partition to a live broker with kafka-reassign-partitions.sh. Do not reach for unclean.leader.election.enable=true as a reflex — it trades the outage for silent data loss.

Fix D — persistent case: routing is wrong

If --describe shows a healthy Leader: 1 but your producer still loops, the leader host it's being handed is unreachable from the client. That's an advertised.listeners problem, not a leadership problem:

- KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+ KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092

The client resolves the advertised leader host directly; if it can't route there, every produce after metadata refresh fails even though the cluster is healthy.

6. Best Practices & The Better Design

Provision topics as code, never by producing to them. The correct-by-construction setup declares topics before any client writes. In Spring, register them as beans and let KafkaAdmin create them at startup:

@Configuration
public class TopicConfig {
    @Bean
    public KafkaAdmin.NewTopics topics() {
        return new KafkaAdmin.NewTopics(
            TopicBuilder.name("orders").partitions(3).replicas(3)
                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")
                .build()
        );
    }
}

KafkaAdmin blocks the context until topics exist, so by the time a @KafkaListener or KafkaTemplate runs, leaders are elected. In production, prefer Strimzi KafkaTopic CRDs or Terraform so topic config lives in review, not in application startup timing.

Give restarts a real replication factor. RF=1 makes every broker restart a guaranteed leaderless gap for its partitions. RF=3 with min.insync.replicas=2 lets a follower take over in milliseconds, so a rolling restart produces at most a flicker of WARNs instead of a stall. Restart brokers one at a time and wait for ISR to fully recover between steps.

Set client timeouts that match reality. Keep max.block.ms generous enough to swallow a normal election (the 60s default is fine; don't drop it to 1–2s in code that runs at startup), rely on 3.0+ default idempotent-producer retry behavior, and let delivery.timeout.ms be the single number that defines "give up." Don't paper over a broken cluster with a huge timeout — fix the ISR.

7. How to Prevent It Long-Term

Alert on the cluster conditions that make LEADER_NOT_AVAILABLE persistent rather than transient:

• kafka.controller:type=KafkaController,name=OfflinePartitionsCount > 0 — partitions with no leader. This is the direct signal.
• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions > 0 — ISR shrinking toward a leaderless state.
• kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs — spikes during restarts; sustained non-zero means flapping.
• Producer client metric record-error-rate > 0 — the WARNs turned into real failures.

Team conventions that prevent it:

• auto.create.topics.enable=false on every non-dev cluster; topics via IaC only.
• Rolling-restart automation that gates each broker on UnderReplicatedPartitions == 0 before proceeding.
• A Testcontainers integration test that creates a topic, restarts the broker mid-produce, and asserts zero lost records on RF≥3 — this catches routing and timeout regressions before prod.
• Treat a LEADER_NOT_AVAILABLE WARN in CI logs that resolves as noise to filter, not a failure to chase; wire the TimeoutException as the thing that fails the build.

8. Key Takeaways

• LEADER_NOT_AVAILABLE (code 5) is retriable: partition exists, no leader right now. It's not UNKNOWN_TOPIC_OR_PARTITION (doesn't exist) or NOT_LEADER_OR_FOLLOWER (wrong broker).
• A brief burst on first-produce or during a restart is normal and self-heals; only a burst that ends in TimeoutException: Topic ... not present in metadata after 60000 ms is a real problem.
• The dev fix is to stop racing auto-creation: auto.create.topics.enable=false plus explicit topic provisioning and a sane max.block.ms.
• The persistent fix is a cluster fix: restore an electable replica (Leader: none) or correct advertised.listeners (leader reachable but unroutable).
• Alert on OfflinePartitionsCount and UnderReplicatedPartitions; RF=3 + min.insync.replicas=2 turns restart stalls into a flicker.