Kafka NotEnoughReplicasException: Fix NOT_ENOUGH_REPLICAS Errors

Your producer callback (or future.get()) blows up with:

org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

Before that surfaces, the producer log spams retries:

WARN [Producer clientId=payments-producer] Got error produce response with correlation id 4721
on topic-partition payments-0, retrying (2147483645 attempts left). Error: NOT_ENOUGH_REPLICAS

And on the partition leader's broker log:

ERROR [ReplicaManager broker=1] Error processing append operation on partition payments-0 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(1) for partition payments-0
is insufficient to satisfy the min.isr requirement of 2 for partition payments-0

If retries never succeed, the application-visible failure is often the less obvious:

org.apache.kafka.common.errors.TimeoutException: Expiring 12 record(s) for payments-0:120000 ms has passed since batch creation

— because NOT_ENOUGH_REPLICAS is retriable and the producer keeps retrying until delivery.timeout.ms (default 120000 ms) expires. If you're only looking at the exception your app throws, you'll chase a "timeout" that is actually an ISR problem. Always check the WARN lines above it.

There's a sibling you may also see: NotEnoughReplicasAfterAppendException (NOT_ENOUGH_REPLICAS_AFTER_APPEND) — the leader appended the batch locally, then noticed the ISR was too small. Also retriable, and the retry can write the batch twice unless idempotence is on (it is by default since Kafka 3.0).

Typical setup: Kafka 3.x/4.x (KRaft), kafka-clients 3.8.x, Spring Kafka 3.2.x, acks=all, replication factor 3, min.insync.replicas=2 — and one or more brokers down, restarting, or lagging. Also extremely common on managed Kafka (MSK, Confluent Cloud) where the cluster default min.insync.replicas=2 collides with a carelessly created --replication-factor 1 topic.

2. How to Reproduce It (step-by-step)

Three-broker KRaft cluster, no ZooKeeper:

# docker-compose.yml
x-kafka-common: &kafka-common
  image: apache/kafka:3.8.1
  environment: &kafka-env
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    KAFKA_MIN_INSYNC_REPLICAS: 2
    CLUSTER_ID: 5L6g3nShT-eMCtK--X86sw

services:
  kafka-1:
    <<: *kafka-common
    ports: ["19092:19092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 1
      KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:19092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:9092,EXTERNAL://localhost:19092
  kafka-2:
    <<: *kafka-common
    ports: ["29092:29092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 2
      KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:29092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-2:9092,EXTERNAL://localhost:29092
  kafka-3:
    <<: *kafka-common
    ports: ["39092:39092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 3
      KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:39092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-3:9092,EXTERNAL://localhost:39092

docker compose up -d

# Topic: RF=3, min.insync.replicas=2
docker exec -it $(docker ps -qf name=kafka-1) \
  /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 \
  --create --topic payments --partitions 3 --replication-factor 3 \
  --config min.insync.replicas=2

Now kill two brokers so every partition's ISR shrinks to 1:

docker compose stop kafka-2 kafka-3

Produce with acks=all:

docker exec -it $(docker ps -qf name=kafka-1) \
  /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:19092 \
  --topic payments --request-required-acks all
> test-message

Within ~30 seconds (replica.lag.time.max.ms, the ISR shrink window) the console producer starts logging NOT_ENOUGH_REPLICAS retries, and eventually the TimeoutException. The equivalent Java producer:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:19092");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

try (var producer = new KafkaProducer<String, String>(props)) {
    producer.send(new ProducerRecord<>("payments", "k1", "v1"), (md, ex) -> {
        if (ex != null) ex.printStackTrace(); // NotEnoughReplicasException or TimeoutException
    });
    producer.flush();
}

Environment-specific triggers:

• Only with acks=all (-1). With acks=1 the same produce succeeds — that's the trap, not a fix (see below).
• The single-broker variant: local docker compose with one broker whose image sets min.insync.replicas=2 cluster-wide, or a managed cluster with default 2 plus a topic created with --replication-factor 1. ISR can never exceed 1, so every acks=all write fails, forever, with a perfectly healthy cluster.
• Rolling restarts with min.insync.replicas equal to the replication factor (e.g. RF=3, min.isr=3): taking down any one broker breaks writes to every partition it hosts.

Confirm the state at any time:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 \
  --describe --under-min-isr-partitions

3. Why It Happens — Surface Level

The partition leader received your batch, checked the partition's current in-sync replica set, and found it smaller than min.insync.replicas. Because you asked for acks=all — "don't ack until every in-sync replica has this" — the broker refuses the write outright rather than accept data with weaker durability than you demanded.

Two configs interact and only two: min.insync.replicas (broker default, overridable per topic) and the producer's acks. The check fires only when acks=all. Replication factor determines how many replicas could be in sync; ISR is how many currently are.

4. Why It Happens — Under the Hood

Each partition leader maintains the ISR: the replicas that are caught up to the leader's log end offset within replica.lag.time.max.ms (default 30000 ms since Kafka 2.5/KIP-537; 10000 ms before that). A follower that hasn't fetched to the end of the log within that window gets removed — the leader (in KRaft, via an AlterPartition request to the controller) shrinks the ISR. You'll see ISR shrink lines in the broker log and IsrShrinksPerSec tick up.

On produce with acks=-1, the leader's append path (ReplicaManager.appendRecords → Partition.appendRecordsToLeader) checks isrSize < minIsr before appending and throws NotEnoughReplicasException. The high watermark — the offset up to which consumers can read — only advances when every ISR member has replicated the record, so a shrunken ISR also means acks are gated on fewer copies. min.insync.replicas exists precisely to put a floor under that: with RF=3 and min.isr=2, a committed record is guaranteed on at least 2 brokers, so losing one broker never loses acked data.

The race variant: the ISR check passes, the leader appends, and the ISR shrinks before enough followers fetch the record. delayedProduce completion then fails with NOT_ENOUGH_REPLICAS_AFTER_APPEND. The record is in the leader's log; the producer retries and appends it again. Without idempotence that's a duplicate — one of the reasons enable.idempotence=true became the default in 3.0.

Why the error is transient in healthy clusters: broker bounces during rolling restarts and follower GC pauses shrink the ISR for seconds. NOT_ENOUGH_REPLICAS is retriable, so the producer's retry loop (bounded by delivery.timeout.ms, not the effectively-infinite retries) usually rides it out invisibly. It becomes an incident when the ISR stays small: a dead broker, a follower that can't keep up with produce throughput, or saturated replication fetchers (num.replica.fetchers, default 1).

And the misconfiguration case has no race at all — RF=1 with min.isr=2 is arithmetic that can never work. Kafka does not validate min.insync.replicas <= replication factor at topic creation.

5. The Fix

Case A: broker(s) down or lagging — restore the ISR. This is an availability incident, not a config problem. Restart the dead broker; check for disk full, long GC, or replication fetcher saturation on lagging followers:

docker compose start kafka-2 kafka-3
# watch the ISR recover
watch /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 \
  --describe --topic payments

Don't touch configs to "fix" a dead broker. Lowering durability to route around a hardware failure is how acked data gets lost.

Case B: min.insync.replicas exceeds what the topic can satisfy. Fix the arithmetic. For an RF=3 topic that was created (or defaulted) to min.isr=3:

- min.insync.replicas=3   # any single broker restart blocks writes
+ min.insync.replicas=2   # tolerates one broker down

/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:19092 \
  --alter --entity-type topics --entity-name payments \
  --add-config min.insync.replicas=2

Takes effect immediately, no restart.

Case C: RF=1 topic on a cluster with default min.isr=2 (the classic managed-Kafka/POC trap). Raise the replication factor — you cannot do it with kafka-topics.sh --alter; it requires a reassignment:

// increase-rf.json
{"version": 1, "partitions": [
  {"topic": "payments", "partition": 0, "replicas": [1, 2, 3]},
  {"topic": "payments", "partition": 1, "replicas": [2, 3, 1]},
  {"topic": "payments", "partition": 2, "replicas": [3, 1, 2]}
]}

/opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server localhost:19092 \
  --reassignment-json-file increase-rf.json --execute

For a throwaway local topic, deleting and recreating with --replication-factor 3 is faster.

What NOT to do: drop to acks=1 to make the error disappear:

- props.put(ProducerConfig.ACKS_CONFIG, "all");
+ props.put(ProducerConfig.ACKS_CONFIG, "1");   // error gone, durability gone

This "works" because min.insync.replicas is only enforced for acks=all. You've silently converted rejected writes into writes that live on one broker. If that broker dies before followers catch up, the data is gone and the leader election hands out a log without it. For anything that matters — payments, ledgers, events feeding downstream state — this is the wrong trade. It also disables idempotence, which requires acks=all.

For POC/dev where you genuinely don't care, set the cluster's expectations honestly instead: one broker, min.insync.replicas=1, RF=1, and keep acks=all so your code doesn't change between laptop and prod.

6. Best Practices & The Better Design

The design that avoids this class of problem is boring and well known — the incidents come from deviating from it on one axis:

# Broker (server.properties) — cluster-wide defaults
default.replication.factor=3
min.insync.replicas=2
auto.create.topics.enable=false          # no accidental RF=1 topics
unclean.leader.election.enable=false     # never elect an out-of-sync leader

# Producer — durable by default (Kafka >= 3.0 defaults shown explicitly)
acks=all
enable.idempotence=true
delivery.timeout.ms=120000
max.in.flight.requests.per.connection=5

Spring Boot equivalent:

spring:
  kafka:
    producer:
      acks: all
      properties:
        enable.idempotence: true
        delivery.timeout.ms: 120000

The invariant to enforce in code review and topic-provisioning tooling: RF − min.isr ≥ 1, i.e. you can lose (RF − min.isr) brokers and still take writes. RF=3/min.isr=2 tolerates one broker down — exactly what a rolling restart needs. If you need to survive two concurrent failures, that's RF=4/min.isr=2 or RF=5/min.isr=3, not RF=3/min.isr=3.

Handle the residual failure honestly in the producer path. NotEnoughReplicasException after delivery.timeout.ms means the cluster could not durably take your write — treat it like a database outage, not a log-and-drop:

producer.send(record, (md, ex) -> {
    if (ex instanceof TimeoutException || ex instanceof NotEnoughReplicasException) {
        // durable-write failure: park to outbox/local queue, trip circuit breaker, alert
        outbox.persist(record);
        meterRegistry.counter("producer.durability.failure").increment();
    }
});

Rack awareness (broker.rack + the rack-aware replica assignment that's default when racks are set) makes RF=3/min.isr=2 survive an AZ outage rather than just a broker outage — with all three replicas in one rack, min.isr=2 protects you from nothing an AZ failure can cause.

7. How to Prevent It Long-Term

Monitor the leading indicators, not the exception. By the time producers throw, you've been under-min-ISR for up to two minutes of retries. Alert on the broker JMX metrics:

• kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount — page immediately if > 0; writes are failing.
• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions — warn if > 0 for more than a few minutes; you're one failure away from the pager.
• kafka.server:type=ReplicaManager,name=IsrShrinksPerSec vs IsrExpandsPerSec — sustained shrink without matching expand means a follower is falling behind (GC, disk, network, fetcher saturation).
• kafka.controller:type=KafkaController,name=OfflinePartitionsCount — should be 0, always.

Validate topics in CI/CD. If topics are provisioned via Terraform/GitOps or an admin-client service, assert replicationFactor >= minInsyncReplicas + 1 at plan time and fail the pipeline. This single check eliminates Case C permanently.

Test rolling restarts before production does it for you. A staging chaos test — restart one broker while producers run at production throughput with acks=all — proves your RF/min.isr arithmetic and shows whether producer retries absorb the shrink window. If that test produces client-visible errors, production rolling upgrades will too.

Capacity-plan replication. Followers replicate everything the leader takes. If produce throughput grows past what num.replica.fetchers=1 can pull, followers lag, ISR shrinks under load, and you get NOT_ENOUGH_REPLICAS exactly at peak traffic. Raise num.replica.fetchers and watch follower fetch lag as part of load testing.

8. Key Takeaways

• NotEnoughReplicasException = ISR size < min.insync.replicas on an acks=all write. It's a durability guarantee working as designed, not a bug.
• It's retriable — the app-visible error is often TimeoutException: Expiring N record(s) after 120 s of hidden NOT_ENOUGH_REPLICAS retries. Read the WARN lines, not just the exception.
• Enforce RF − min.isr ≥ 1 (RF=3/min.isr=2 is the standard). RF=1 + min.isr=2 fails forever; RF=3 + min.isr=3 breaks every rolling restart. Kafka won't validate this for you.
• Never "fix" it with acks=1 — that trades rejected writes for silent data loss and disables idempotence.
• Alert on UnderMinIsrPartitionCount > 0 (page) and UnderReplicatedPartitions > 0 (warn) — they fire before your producers do.