Kafka InvalidReplicationFactorException: Larger Than Available Brokers

1. The Error

Creating a topic against a single-broker cluster:

org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

With newer kafka-topics.sh (KRaft-era, AdminClient-based), the CLI prints the controller's message instead:

Error while executing topic command : Unable to replicate the partition 3 time(s): The target replication factor of 3 cannot be reached because only 1 broker(s) are registered.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 3 time(s): The target replication factor of 3 cannot be reached because only 1 broker(s) are registered.

From Spring Boot, it surfaces at startup when KafkaAdmin tries to create your NewTopic beans:

org.springframework.kafka.KafkaException: Failed to create topics
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 3 time(s): The target replication factor of 3 cannot be reached because only 1 broker(s) are registered.

And there's a sneakier variant with no exception in your application at all: your consumer hangs forever, kafka-consumer-groups.sh can't describe anything, and the broker log shows the same InvalidReplicationFactorException — because the broker can't create __consumer_offsets with its default replication factor of 3. Client side you'll only see this on repeat:

[Consumer clientId=consumer-orders-1, groupId=orders] Group coordinator lookup failed: The coordinator is not available.
[Consumer clientId=consumer-orders-1, groupId=orders] Coordinator discovery failed, refreshing metadata

Typical setup: Kafka 3.x/4.x in KRaft mode, single broker in Docker (apache/kafka:3.8.0, Bitnami, or a hand-rolled compose file), kafka-clients 3.x, Spring Kafka 3.x. This is overwhelmingly a local/POC error — but the __consumer_offsets variant also bites real clusters during cold-start DR scenarios when brokers register slower than clients connect.

2. How to Reproduce It (step-by-step)

Minimal single-broker KRaft cluster, without the internal-topic overrides:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.8.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      # NOTE: no KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR override — defaults to 3

docker compose up -d

Trigger 1 — explicit topic creation with RF > broker count:

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 3

Fails immediately with the "only 1 broker(s) are registered" message.

Trigger 2 — the hidden one. Create a valid topic, then try to consume with a group:

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders --group orders --from-beginning

The consumer hangs. Check the broker log:

docker logs kafka 2>&1 | grep -i "InvalidReplicationFactor"

You'll find the broker failing to auto-create __consumer_offsets (RF 3, one broker). Same story for initTransactions() hanging on a transactional producer — that's __transaction_state failing creation with transaction.state.log.replication.factor=3.

Environment-specific notes: the official single-node quickstart compose files set these overrides for you, which is exactly why the error appears the moment someone writes their own compose file or trims an example. Confluent's cp-kafka image has the same defaults and the same fix (env var names identical apart from the image's prefix conventions).

3. Why It Happens — Surface Level

Replication factor is the number of distinct brokers that must each hold a full copy of every partition. Kafka never places two replicas of the same partition on one broker — that would be a replica in name only. So RF 3 requires at least 3 registered, unfenced brokers. You have 1. The controller rejects the assignment outright with InvalidReplicationFactorException — it fails fast rather than creating an under-replicated topic that can never heal.

The hidden variant is the same arithmetic applied to Kafka's own internal topics: __consumer_offsets defaults to offsets.topic.replication.factor=3 and __transaction_state to transaction.state.log.replication.factor=3. On a single broker, neither can be created — so group coordination and transactions never come up.

4. Why It Happens — Under the Hood

In KRaft mode, topic creation is handled by ReplicationControlManager on the active controller. For each partition it asks the replica placer (StripedReplicaPlacer) for an assignment across the cluster's registered brokers. The placer works from the controller's broker registration table — brokers that have registered via heartbeat and are not fenced. Fenced brokers (registered once, but currently failing heartbeats) are excluded from placement, which is why a cluster that had 3 brokers can still throw "only 1 broker(s) are registered" mid-outage or during a rolling restart of a small cluster. If count(usable brokers) < replicationFactor, placement is impossible and the controller returns INVALID_REPLICATION_FACTOR (error code 38) to the AdminClient. Same error code, by the way, if you pass --replication-factor 0 or -1 without letting the broker default apply. In legacy ZooKeeper mode the check lived in AdminUtils.assignReplicasToBrokers, producing the shorter "Replication factor: 3 larger than available brokers: 1" wording — that's why two phrasings of this error circulate.

The __consumer_offsets path is lazier and therefore more confusing. The offsets topic is not created at broker startup; it's created on the first FindCoordinator request that needs it. The group coordinator's partition for your group is hash(group.id) % offsets.topic.num.partitions, and until the topic exists there is no coordinator to find. Since KIP-115 (Kafka 0.11), offsets.topic.replication.factor is enforced strictly: the broker will not silently create the topic with whatever brokers happen to be alive (which old versions did, permanently pinning your offsets topic at RF 1 — a production time bomb). Instead, creation fails, the client gets COORDINATOR_NOT_AVAILABLE, and the consumer retries forever: FindCoordinator → error → backoff → retry. Nothing ever escalates to your application code because COORDINATOR_NOT_AVAILABLE is retriable by design. The result is the worst kind of failure: an infinite silent hang whose actual exception only exists in the broker log. __transaction_state behaves identically for initTransactions(), with the extra trap that transaction.state.log.min.isr=2 must also be lowered on a single broker or transactions will come up and then refuse writes.

5. The Fix

Dev / single-broker Docker: align every RF-related default with your broker count.

     environment:
       KAFKA_NODE_ID: 1
       KAFKA_PROCESS_ROLES: broker,controller
       KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
       KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
       KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
       KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_DEFAULT_REPLICATION_FACTOR: 1

Then recreate the environment (the failed internal-topic state is harmless, but if you've been experimenting, start clean):

docker compose down -v && docker compose up -d

And create topics with --replication-factor 1 — or let them inherit default.replication.factor.

Spring Boot topic beans, dev profile:

 @Bean
 public NewTopic ordersTopic() {
-    return TopicBuilder.name("orders").partitions(3).replicas(3).build();
+    return TopicBuilder.name("orders")
+            .partitions(3)
+            .replicas(replicationFactor)   // profile property: 1 dev, 3 prod
+            .build();
 }

# application-dev.properties
app.kafka.replication-factor=1
# application-prod.properties
app.kafka.replication-factor=3

Production: never lower the RF — raise the broker count or fix registration. If a real cluster throws this, the question is why the controller sees fewer brokers than you think it has:

# KRaft: what does the controller actually see?
/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server broker1:9092 | grep -c "id:"
# Or, on 3.7+:
/opt/kafka/bin/kafka-cluster.sh cluster-id --bootstrap-server broker1:9092

Fenced brokers, brokers pointed at the wrong controller quorum, or a partially completed scale-down are the usual suspects. Fix registration; do not "fix" the RF.

Which fix when: single-broker POC → the compose diff above, full stop. Shared dev cluster with 3 nodes → keep RF 3 everywhere, including internal topics, so dev behaves like prod. Production → RF 3, min.insync.replicas=2, and treat this exception as a broker-registration incident, not a config knob.

6. Best Practices & The Better Design

The class of bug here is environment-dependent replication arithmetic hardcoded in the wrong place. The better design has two parts.

First, make dev topology explicit and version-controlled — one known-good single-broker compose file with all four RF-related overrides, owned by the team, instead of every engineer trimming a random example. If your integration tests use Testcontainers, this is already handled: KafkaContainer sets the internal-topic RFs to 1 for you, which is one more reason to prefer it over a shared "dev cluster" for CI:

// kafka-clients 3.8.x, testcontainers 1.20.x
@Container
static final KafkaContainer kafka =
        new KafkaContainer(DockerImageName.parse("apache/kafka:3.8.0"));
// internal topic RFs = 1 out of the box; no InvalidReplicationFactorException possible

Second, treat replication factor as deployment configuration, never as code. No literal 3 in TopicBuilder, no literal 3 in a kafka-topics.sh runbook. Topics-as-code tooling (Strimzi KafkaTopic CRs, Terraform kafka_topic, JulieOps) keeps per-environment RF in the environment's own manifest, and the PR diff shows exactly which environment gets which RF. If you must create topics from application code, source the RF from config as shown above — and set spring.kafka.admin.fail-fast=true so a misconfigured RF kills the app at startup with the real exception instead of surfacing later as a mystery hang.

7. How to Prevent It Long-Term

Monitor broker registration, not just broker process health: in KRaft, alert when the controller's active broker count (kafka.controller:type=KafkaController,name=ActiveBrokerCount, with FencedBrokerCount alongside) drops below the highest RF you use. That catches the production version of this error — fenced or unregistered brokers — before topic creation starts failing. Add a CI gate that lints topic manifests: rf <= expected broker count per environment, and rf >= 3 with min.insync.replicas = rf - 1 for prod. Chaos-test small clusters with a rolling restart while a deploy creates topics; a 3-broker cluster creating RF-3 topics has zero headroom during any single-broker restart, and this exception is exactly how that surfaces. Finally, keep one canonical local-dev compose file in your platform repo and delete the copies — nearly every occurrence of this error in a POC traces back to a hand-edited compose file missing KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1.

8. Key Takeaways

• InvalidReplicationFactorException means the controller has fewer registered, unfenced brokers than the requested RF — Kafka fails fast instead of creating a topic that can never be fully replicated.
• A consumer that hangs forever with "Group coordinator lookup failed: The coordinator is not available" on a fresh single-broker setup is the same error in disguise: __consumer_offsets can't be created at its default RF of 3. The exception is only in the broker log.
• Single-broker dev needs all four: offsets.topic.replication.factor=1, transaction.state.log.replication.factor=1, transaction.state.log.min.isr=1, default.replication.factor=1.
• In production, this exception is a broker-registration incident (fenced brokers, wrong quorum, scale-down gone wrong) — fix registration, never lower the RF.
• Keep RF out of code: per-environment config or topics-as-code, with a CI check that RF fits the target cluster.