> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka NotEnoughReplicasException: Fix NOT_ENOUGH_REPLICAS Errors - URL: https://www.ggorantala.dev/kafka-notenoughreplicasexception-min-insync-replicas/ - Published: 2026-03-30T10:19:00.000Z - Updated: 2026-08-29T08:26:55.000Z - Description: Kafka NotEnoughReplicasException: messages rejected since there are fewer in-sync replicas than required. Fix min.insync.replicas vs acks=all, fast. - Author: Gopi Gorantala - Tags: kafka-errors, notenoughreplicasexception, min-insync-replicas, kafka-producer, kafka-replication, spring-kafka, Java Your producer callback (or `future.get()`) blows up with: ```text org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required. ``` Before that surfaces, the producer log spams retries: ```text WARN [Producer clientId=payments-producer] Got error produce response with correlation id 4721 on topic-partition payments-0, retrying (2147483645 attempts left). Error: NOT_ENOUGH_REPLICAS ``` And on the partition leader's broker log: ```text ERROR [ReplicaManager broker=1] Error processing append operation on partition payments-0 (kafka.server.ReplicaManager) org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(1) for partition payments-0 is insufficient to satisfy the min.isr requirement of 2 for partition payments-0 ``` If retries never succeed, the application-visible failure is often the less obvious: ```text org.apache.kafka.common.errors.TimeoutException: Expiring 12 record(s) for payments-0:120000 ms has passed since batch creation ``` — because `NOT_ENOUGH_REPLICAS` is retriable and the producer keeps retrying until `delivery.timeout.ms` (default 120000 ms) expires. If you're only looking at the exception your app throws, you'll chase a "timeout" that is actually an ISR problem. Always check the WARN lines above it. There's a sibling you may also see: `NotEnoughReplicasAfterAppendException` (`NOT_ENOUGH_REPLICAS_AFTER_APPEND`) — the leader appended the batch locally, *then* noticed the ISR was too small. Also retriable, and the retry can write the batch twice unless idempotence is on (it is by default since Kafka 3.0). **Typical setup:** Kafka 3.x/4.x (KRaft), `kafka-clients` 3.8.x, Spring Kafka 3.2.x, `acks=all`, replication factor 3, `min.insync.replicas=2` — and one or more brokers down, restarting, or lagging. Also extremely common on managed Kafka (MSK, Confluent Cloud) where the cluster default `min.insync.replicas=2` collides with a carelessly created `--replication-factor 1` topic. ## 2\. How to Reproduce It (step-by-step) Three-broker KRaft cluster, no ZooKeeper: ```yaml # docker-compose.yml x-kafka-common: &kafka-common image: apache/kafka:3.8.1 environment: &kafka-env KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_MIN_INSYNC_REPLICAS: 2 CLUSTER_ID: 5L6g3nShT-eMCtK--X86sw services: kafka-1: <<: *kafka-common ports: ["19092:19092"] environment: <<: *kafka-env KAFKA_NODE_ID: 1 KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:19092 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:9092,EXTERNAL://localhost:19092 kafka-2: <<: *kafka-common ports: ["29092:29092"] environment: <<: *kafka-env KAFKA_NODE_ID: 2 KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:29092 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-2:9092,EXTERNAL://localhost:29092 kafka-3: <<: *kafka-common ports: ["39092:39092"] environment: <<: *kafka-env KAFKA_NODE_ID: 3 KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:39092 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-3:9092,EXTERNAL://localhost:39092 ``` ```bash docker compose up -d # Topic: RF=3, min.insync.replicas=2 docker exec -it $(docker ps -qf name=kafka-1) \ /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 \ --create --topic payments --partitions 3 --replication-factor 3 \ --config min.insync.replicas=2 ``` Now kill two brokers so every partition's ISR shrinks to 1: ```bash docker compose stop kafka-2 kafka-3 ``` Produce with `acks=all`: ```bash docker exec -it $(docker ps -qf name=kafka-1) \ /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:19092 \ --topic payments --request-required-acks all > test-message ``` Within \~30 seconds (`replica.lag.time.max.ms`, the ISR shrink window) the console producer starts logging `NOT_ENOUGH_REPLICAS` retries, and eventually the `TimeoutException`. The equivalent Java producer: ```java Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:19092"); props.put(ProducerConfig.ACKS_CONFIG, "all"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); try (var producer = new KafkaProducer(props)) { producer.send(new ProducerRecord<>("payments", "k1", "v1"), (md, ex) -> { if (ex != null) ex.printStackTrace(); // NotEnoughReplicasException or TimeoutException }); producer.flush(); } ``` **Environment-specific triggers:** - Only with `acks=all` (`-1`). With `acks=1` the same produce *succeeds* — that's the trap, not a fix (see below). - The single-broker variant: local `docker compose` with one broker whose image sets `min.insync.replicas=2` cluster-wide, or a managed cluster with default `2` plus a topic created with `--replication-factor 1`. ISR can never exceed 1, so every `acks=all` write fails, forever, with a perfectly healthy cluster. - Rolling restarts with `min.insync.replicas` equal to the replication factor (e.g. RF=3, min.isr=3): taking down any one broker breaks writes to every partition it hosts. Confirm the state at any time: ```bash /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 \ --describe --under-min-isr-partitions ``` ## 3\. Why It Happens — Surface Level The partition leader received your batch, checked the partition's current in-sync replica set, and found it smaller than `min.insync.replicas`. Because you asked for `acks=all` — "don't ack until every in-sync replica has this" — the broker refuses the write outright rather than accept data with weaker durability than you demanded. Two configs interact and only two: `min.insync.replicas` (broker default, overridable per topic) and the producer's `acks`. The check fires **only** when `acks=all`. Replication factor determines how many replicas *could* be in sync; ISR is how many currently *are*. ## 4\. Why It Happens — Under the Hood Each partition leader maintains the ISR: the replicas that are caught up to the leader's log end offset within `replica.lag.time.max.ms` (default 30000 ms since Kafka 2.5/KIP-537; 10000 ms before that). A follower that hasn't fetched to the end of the log within that window gets removed — the leader (in KRaft, via an `AlterPartition` request to the controller) shrinks the ISR. You'll see `ISR shrink` lines in the broker log and `IsrShrinksPerSec` tick up. On produce with `acks=-1`, the leader's append path (`ReplicaManager.appendRecords` → `Partition.appendRecordsToLeader`) checks `isrSize < minIsr` *before* appending and throws `NotEnoughReplicasException`. The high watermark — the offset up to which consumers can read — only advances when every ISR member has replicated the record, so a shrunken ISR also means acks are gated on fewer copies. `min.insync.replicas` exists precisely to put a floor under that: with RF=3 and min.isr=2, a committed record is guaranteed on at least 2 brokers, so losing one broker never loses acked data. The race variant: the ISR check passes, the leader appends, and the ISR shrinks before enough followers fetch the record. `delayedProduce` completion then fails with `NOT_ENOUGH_REPLICAS_AFTER_APPEND`. The record *is* in the leader's log; the producer retries and appends it again. Without idempotence that's a duplicate — one of the reasons `enable.idempotence=true` became the default in 3.0. Why the error is transient in healthy clusters: broker bounces during rolling restarts and follower GC pauses shrink the ISR for seconds. `NOT_ENOUGH_REPLICAS` is retriable, so the producer's retry loop (bounded by `delivery.timeout.ms`, not the effectively-infinite `retries`) usually rides it out invisibly. It becomes an incident when the ISR *stays* small: a dead broker, a follower that can't keep up with produce throughput, or saturated replication fetchers (`num.replica.fetchers`, default 1). And the misconfiguration case has no race at all — RF=1 with min.isr=2 is arithmetic that can never work. Kafka does not validate `min.insync.replicas <= replication factor` at topic creation. ## 5\. The Fix **Case A: broker(s) down or lagging — restore the ISR.** This is an availability incident, not a config problem. Restart the dead broker; check for disk full, long GC, or replication fetcher saturation on lagging followers: ```bash docker compose start kafka-2 kafka-3 # watch the ISR recover watch /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 \ --describe --topic payments ``` Don't touch configs to "fix" a dead broker. Lowering durability to route around a hardware failure is how acked data gets lost. **Case B: min.insync.replicas exceeds what the topic can satisfy.** Fix the arithmetic. For an RF=3 topic that was created (or defaulted) to min.isr=3: ```diff - min.insync.replicas=3 # any single broker restart blocks writes + min.insync.replicas=2 # tolerates one broker down ``` ```bash /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:19092 \ --alter --entity-type topics --entity-name payments \ --add-config min.insync.replicas=2 ``` Takes effect immediately, no restart. **Case C: RF=1 topic on a cluster with default min.isr=2** (the classic managed-Kafka/POC trap). Raise the replication factor — you cannot do it with `kafka-topics.sh --alter`; it requires a reassignment: ```json // increase-rf.json {"version": 1, "partitions": [ {"topic": "payments", "partition": 0, "replicas": [1, 2, 3]}, {"topic": "payments", "partition": 1, "replicas": [2, 3, 1]}, {"topic": "payments", "partition": 2, "replicas": [3, 1, 2]} ]} ``` ```bash /opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server localhost:19092 \ --reassignment-json-file increase-rf.json --execute ``` For a throwaway local topic, deleting and recreating with `--replication-factor 3` is faster. **What NOT to do:** drop to `acks=1` to make the error disappear: ```diff - props.put(ProducerConfig.ACKS_CONFIG, "all"); + props.put(ProducerConfig.ACKS_CONFIG, "1"); // error gone, durability gone ``` This "works" because `min.insync.replicas` is only enforced for `acks=all`. You've silently converted rejected writes into writes that live on one broker. If that broker dies before followers catch up, the data is gone and the leader election hands out a log without it. For anything that matters — payments, ledgers, events feeding downstream state — this is the wrong trade. It also disables idempotence, which requires `acks=all`. For POC/dev where you genuinely don't care, set the *cluster's* expectations honestly instead: one broker, `min.insync.replicas=1`, RF=1, and keep `acks=all` so your code doesn't change between laptop and prod. ## 6\. Best Practices & The Better Design The design that avoids this class of problem is boring and well known — the incidents come from deviating from it on one axis: ```properties # Broker (server.properties) — cluster-wide defaults default.replication.factor=3 min.insync.replicas=2 auto.create.topics.enable=false # no accidental RF=1 topics unclean.leader.election.enable=false # never elect an out-of-sync leader ``` ```properties # Producer — durable by default (Kafka >= 3.0 defaults shown explicitly) acks=all enable.idempotence=true delivery.timeout.ms=120000 max.in.flight.requests.per.connection=5 ``` Spring Boot equivalent: ```yaml spring: kafka: producer: acks: all properties: enable.idempotence: true delivery.timeout.ms: 120000 ``` The invariant to enforce in code review and topic-provisioning tooling: **RF − min.isr ≥ 1**, i.e. you can lose (RF − min.isr) brokers and still take writes. RF=3/min.isr=2 tolerates one broker down — exactly what a rolling restart needs. If you need to survive two concurrent failures, that's RF=4/min.isr=2 or RF=5/min.isr=3, not RF=3/min.isr=3. Handle the residual failure honestly in the producer path. `NotEnoughReplicasException` after `delivery.timeout.ms` means the cluster could not durably take your write — treat it like a database outage, not a log-and-drop: ```java producer.send(record, (md, ex) -> { if (ex instanceof TimeoutException || ex instanceof NotEnoughReplicasException) { // durable-write failure: park to outbox/local queue, trip circuit breaker, alert outbox.persist(record); meterRegistry.counter("producer.durability.failure").increment(); } }); ``` Rack awareness (`broker.rack` \+ the rack-aware replica assignment that's default when racks are set) makes RF=3/min.isr=2 survive an AZ outage rather than just a broker outage — with all three replicas in one rack, min.isr=2 protects you from nothing an AZ failure can cause. ## 7\. How to Prevent It Long-Term **Monitor the leading indicators, not the exception.** By the time producers throw, you've been under-min-ISR for up to two minutes of retries. Alert on the broker JMX metrics: - `kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount` — page immediately if > 0; writes are failing. - `kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions` — warn if > 0 for more than a few minutes; you're one failure away from the pager. - `kafka.server:type=ReplicaManager,name=IsrShrinksPerSec` vs `IsrExpandsPerSec` — sustained shrink without matching expand means a follower is falling behind (GC, disk, network, fetcher saturation). - `kafka.controller:type=KafkaController,name=OfflinePartitionsCount` — should be 0, always. **Validate topics in CI/CD.** If topics are provisioned via Terraform/GitOps or an admin-client service, assert `replicationFactor >= minInsyncReplicas + 1` at plan time and fail the pipeline. This single check eliminates Case C permanently. **Test rolling restarts before production does it for you.** A staging chaos test — restart one broker while producers run at production throughput with `acks=all` — proves your RF/min.isr arithmetic and shows whether producer retries absorb the shrink window. If that test produces client-visible errors, production rolling upgrades will too. **Capacity-plan replication.** Followers replicate everything the leader takes. If produce throughput grows past what `num.replica.fetchers=1` can pull, followers lag, ISR shrinks under load, and you get `NOT_ENOUGH_REPLICAS` exactly at peak traffic. Raise `num.replica.fetchers` and watch follower fetch lag as part of load testing. ## 8\. Key Takeaways - `NotEnoughReplicasException` \= ISR size < `min.insync.replicas` on an `acks=all` write. It's a durability guarantee working as designed, not a bug. - It's retriable — the app-visible error is often `TimeoutException: Expiring N record(s)` after 120 s of hidden `NOT_ENOUGH_REPLICAS` retries. Read the WARN lines, not just the exception. - Enforce **RF − min.isr ≥ 1** (RF=3/min.isr=2 is the standard). RF=1 + min.isr=2 fails forever; RF=3 + min.isr=3 breaks every rolling restart. Kafka won't validate this for you. - Never "fix" it with `acks=1` — that trades rejected writes for silent data loss and disables idempotence. - Alert on `UnderMinIsrPartitionCount > 0` (page) and `UnderReplicatedPartitions > 0` (warn) — they fire before your producers do.