Kafka TopicExistsException: Why It Happens and How to Fix It

You wired up an AdminClient.createTopics(...) call — or a startup bean that provisions topics — and now the app won't boot. The stack trace ends in TopicExistsException. This is one of the most misunderstood "errors" in Kafka because half the time it is not an error at all: it is Kafka telling you the topic is already there, and your code treating a benign, expected condition as fatal. The other half of the time it is a genuine race between two instances, or a partition-count mismatch masquerading as "already exists."

This article splits the two apart, shows the exact mechanics of the CreateTopics path (KRaft-era), and gives you idempotent provisioning code that never crashes on this again.

1. The Error

The raw exception, as it surfaces from a KafkaFuture in AdminClient:

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists.
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
Caused by: org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists.

Note the wrapping: createTopics(...).all().get() throws ExecutionException, and the real cause — TopicExistsException — is a subclass of ApiException, carrying broker error code 36 (TOPIC_ALREADY_EXISTS). It is not retriable. Retrying will get you the same answer forever.

In Spring Boot, the same condition surfaces during context startup and takes the whole application down:

org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'
...
Caused by: org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists.

And from the CLI:

$ kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
Error while executing topic command : Topic 'orders' already exists.
[2026-07-12 10:14:22,317] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists.
 (kafka.admin.TopicCommand$)

Where it typically shows up:

• Kafka: 3.x / 4.0, KRaft mode. Same code path and error existed under ZooKeeper.
• Clients: kafka-clients AdminClient (any modern version), Spring Kafka KafkaAdmin + NewTopic beans.
• Setup: Docker Compose POCs where the app auto-creates topics on boot and gets restarted; multi-replica deployments where every pod runs the same provisioning code; CI pipelines that re-run topic setup against a persistent cluster.

2. How to Reproduce It

Minimal KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"

docker compose up -d

Create the topic once, then create it again — the second call fails:

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
# Created topic orders.

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
# Error while executing topic command : Topic 'orders' already exists.

The programmatic version (kafka-clients 3.9.0):

try (Admin admin = Admin.create(Map.of(
        AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"))) {

    NewTopic topic = new NewTopic("orders", 3, (short) 1);

    // First call: succeeds.
    admin.createTopics(List.of(topic)).all().get();

    // Second call: throws ExecutionException -> TopicExistsException.
    admin.createTopics(List.of(topic)).all().get();
}

The race variant needs no second manual step — just start two instances of the same provisioning code at once (two pods, or two JVMs), each holding a NewTopic("orders", ...). Both send CreateTopics before either sees the other's result. One wins; the other's KafkaFuture completes with TopicExistsException. In CI this looks flaky: green on a fresh cluster, red on a re-run against a persistent one.

3. Why It Happens — Surface Level

Topic creation is not idempotent by default. CreateTopics means "create a brand-new topic," not "ensure this topic exists." If a topic with that name is already present in cluster metadata, the broker rejects the request with TOPIC_ALREADY_EXISTS. That is the entire surface cause. Your code — or Spring, or the CLI — then propagates that rejection as a fatal exception because it treated a "create" as "create or ignore."

The three concrete triggers:

1. Re-running create logic against a cluster that already has the topic (restarts, CI replays, dev loops).
2. Two instances racing to create the same topic on startup.
3. A partition/RF mismatch — you think you are changing the topic, but CreateTopics can't modify an existing topic; it only ever rejects it.

4. Why It Happens — Under the Hood

In KRaft mode, AdminClient.createTopics sends a CreateTopics request to the active controller (the QuorumController). The controller is the single writer of cluster metadata. It replays the request against the current metadata image: for each topic in the batch it checks whether a TopicRecord with that name already exists in the __cluster_metadata log. If it does, that topic's result in the response is stamped with error code 36 and no TopicRecord is written. The controller processes the batch atomically per-topic, so in a multi-topic request the topics that don't exist are still created; only the colliding ones fail. That is why createTopics(...).values() gives you a per-topic KafkaFuture — you can have partial success.

This design is what makes the race deterministic rather than random. Metadata changes are serialized through the controller's single-threaded event loop. When two instances fire CreateTopics for orders, the controller orders them: the first mutation appends a TopicRecord and the topic now exists in the metadata image; the second is replayed against that updated image and sees the topic, so it returns 36. There is no lock to contend on and no corruption — one create, one rejection, every time. (Under ZooKeeper the same outcome came from a znode existence check under /brokers/topics; the API contract is identical.)

Two subtleties that trip people up:

• CreateTopics never modifies. There is no "upsert." To add partitions you call createPartitions; to change configs you call incrementalAlterConfigs. Handing CreateTopics a NewTopic with a different partition count than the live topic does not reshape it — it just returns TopicExistsException. The mismatch is invisible unless you inspect the live topic yourself.
• validateOnly and defaults. CreateTopicsOptions.validateOnly(true) runs the checks without writing — and a validateOnly create of an existing topic still returns TOPIC_ALREADY_EXISTS. Since KIP-464 (Kafka 2.4) you can pass -1 for partitions and replication factor to accept broker defaults (num.partitions, default.replication.factor), but that only affects creation, not the existence check.

Spring's KafkaAdmin deserves its own note because its behavior is version-dependent. On startup it collects all NewTopic beans and reconciles them: it describes existing topics, creates only the missing ones, and — if a bean declares more partitions than the live topic — issues createPartitions to grow it. So in the common case Spring does not throw TopicExistsException; it swallows the existing-topic case for you. You hit the exception in Spring when reconciliation itself fails — e.g. a bean asks for fewer partitions than exist (illegal, surfaces as InvalidPartitionsException), or you bypass KafkaAdmin and call AdminClient directly, or fatalIfBrokerNotAvailable interacts with a create you issued by hand. If a bare TopicExistsException is taking down your Spring context, something is calling createTopics outside the KafkaAdmin reconciliation path.

5. The Fix

The fix is to make provisioning idempotent: treat TopicExistsException as success, not failure. Do not retry it, and do not let it abort startup.

Before — fatal on a benign condition:

NewTopic topic = new NewTopic("orders", 3, (short) 1);
admin.createTopics(List.of(topic)).all().get();   // throws if topic exists

After — catch and treat as idempotent:

NewTopic topic = new NewTopic("orders", 3, (short) 1);
try {
    admin.createTopics(List.of(topic)).all().get();
    log.info("Created topic {}", topic.name());
} catch (ExecutionException e) {
    if (e.getCause() instanceof TopicExistsException) {
        log.info("Topic {} already exists — skipping", topic.name());
    } else {
        throw e;   // real failure: rethrow
    }
}

For multi-topic batches, unwrap per topic so one existing topic doesn't hide real failures on the others:

CreateTopicsResult result = admin.createTopics(List.of(
        new NewTopic("orders", 3, (short) 1),
        new NewTopic("payments", 3, (short) 1)));

result.values().forEach((name, future) -> {
    try {
        future.get();
        log.info("Created {}", name);
    } catch (ExecutionException e) {
        if (e.getCause() instanceof TopicExistsException) {
            log.info("{} already exists — skipping", name);
        } else {
            throw new RuntimeException("Failed to create " + name, e.getCause());
        }
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
        throw new RuntimeException(e);
    }
});

On the CLI, use --if-not-exists to make create idempotent:

kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic orders --partitions 3 --replication-factor 1 --if-not-exists

If the real problem is a partition-count change, stop reaching for createTopics. Grow partitions explicitly (this only ever increases — you cannot shrink):

admin.createPartitions(Map.of(
        "orders", NewPartitions.increaseTo(6))).all().get();

Which fix, when:

• POC / single instance: the catch-and-log pattern above is enough.
• Multi-instance startup: same catch, but now it is load-bearing — every replica runs it and all but one will legitimately see TopicExistsException. That is expected, not a bug.
• You need a different shape: describe first, then createPartitions / incrementalAlterConfigs. Never assume createTopics reshapes anything.

6. Best Practices & The Better Design

The class of bug here is "application code owns topic lifecycle." App-side auto-provisioning is fine for POCs and convenient in dev, but it forces every instance to race on startup and makes topic shape drift silently.

Better design 1 — let Spring's KafkaAdmin own it. Declare topics as beans and let reconciliation handle existence and growth:

@Configuration
public class KafkaTopicConfig {

    @Bean
    public KafkaAdmin.NewTopics appTopics() {
        return new KafkaAdmin.NewTopics(
            TopicBuilder.name("orders").partitions(3).replicas(1).build(),
            TopicBuilder.name("payments").partitions(3).replicas(1)
                .config(TopicConfig.RETENTION_MS_CONFIG, "604800000")
                .build()
        );
    }
}

KafkaAdmin creates missing topics, grows partitions where a bean asks for more, and — critically — does not throw when the topics already exist. This is the idempotent path you'd otherwise hand-roll.

Better design 2 — topics as code, out of the app entirely. In any real environment, provision topics with a declarative tool that reconciles desired vs actual state: Strimzi KafkaTopic CRDs on Kubernetes, Terraform's Kafka provider, or a gated setup job in CI. The application then assumes its topics exist and never calls createTopics. This removes the startup race, kills the "which instance creates it" question, and makes partition/RF/retention reviewable in a pull request:

# Strimzi KafkaTopic
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 3
  config:
    retention.ms: "604800000"
    min.insync.replicas: "2"

Better design 3 — turn off broker auto-creation. Set auto.create.topics.enable=false on the cluster. This isn't about TopicExistsException directly, but it forces topics to be created deliberately (through one of the paths above) instead of appearing implicitly on first produce/consume — which is what leads to duplicated, racing create logic in the first place.

7. How to Prevent It Long-Term

• Never treat createTopics as create-or-update. Bake the TopicExistsException-is-success rule into your provisioning helper and reuse it everywhere.
• Make provisioning declarative. Strimzi / Terraform / a single CI job as the one owner of topic state. App instances don't create topics.
• Guard partition/config drift in CI. A pipeline check that describes live topics and diffs them against the declared spec catches "someone changed partitions in a NewTopic bean and it silently did nothing."
• Alert on the wrong signals. A steady trickle of TOPIC_ALREADY_EXISTS in controller logs during rollouts is normal (racing replicas). A spike outside deploys, or InvalidPartitionsException, means someone is trying to reshape a topic through the create path — that's the real alertable event.
• Test the race. A Testcontainers test that starts two AdminClients creating the same topic concurrently and asserts exactly one success + one TopicExistsException locks in the idempotent handling so a refactor can't reintroduce the crash.

8. Key Takeaways

• TopicExistsException (code 36, TOPIC_ALREADY_EXISTS) is non-retriable and usually benign — createTopics means "create new," not "ensure exists."
• Unwrap it from ExecutionException per topic and treat it as success; only rethrow the causes that aren't TopicExistsException.
• The KRaft controller serializes creates through a single-writer event loop, so the multi-instance race is deterministic: one create, one rejection — not corruption.
• createTopics never modifies a topic. Use createPartitions to grow and incrementalAlterConfigs to change configs; a partition mismatch shows up as TopicExistsException, not a reshape.
• Best design: don't provision topics from app code. Use Spring KafkaAdmin reconciliation for dev, and Strimzi/Terraform (topics as code) for anything real.