> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka TopicExistsException: Why It Happens and How to Fix It - URL: https://www.ggorantala.dev/kafka-topicexistsexception-fix/ - Published: 2026-06-07T14:23:00.000Z - Updated: 2026-08-29T08:34:36.000Z - Description: Kafka TopicExistsException: Topic 'X' already exists. Why AdminClient, Spring KafkaAdmin, and kafka-topics.sh throw it, and how to handle it idempotently. - Author: Gopi Gorantala - Tags: kafka-errors, TopicExistsException, kafka-adminclient, spring-kafka, kafka-docker, Java, event-streaming You wired up an `AdminClient.createTopics(...)` call — or a startup bean that provisions topics — and now the app won't boot. The stack trace ends in `TopicExistsException`. This is one of the most misunderstood "errors" in Kafka because half the time it is not an error at all: it is Kafka telling you the topic is already there, and your code treating a benign, expected condition as fatal. The other half of the time it is a genuine race between two instances, or a partition-count mismatch masquerading as "already exists." This article splits the two apart, shows the exact mechanics of the CreateTopics path (KRaft-era), and gives you idempotent provisioning code that never crashes on this again. ## 1\. The Error The raw exception, as it surfaces from a `KafkaFuture` in `AdminClient`: ```log java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) Caused by: org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists. ``` Note the wrapping: `createTopics(...).all().get()` throws `ExecutionException`, and the real cause — `TopicExistsException` — is a subclass of `ApiException`, carrying broker error code **36 (`TOPIC_ALREADY_EXISTS`)**. It is **not** retriable. Retrying will get you the same answer forever. In Spring Boot, the same condition surfaces during context startup and takes the whole application down: ```text org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry' ... Caused by: org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists. ``` And from the CLI: ```text $ kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1 Error while executing topic command : Topic 'orders' already exists. [2026-07-12 10:14:22,317] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'orders' already exists. (kafka.admin.TopicCommand$) ``` Where it typically shows up: - **Kafka**: 3.x / 4.0, KRaft mode. Same code path and error existed under ZooKeeper. - **Clients**: `kafka-clients` `AdminClient` (any modern version), Spring Kafka `KafkaAdmin` \+ `NewTopic` beans. - **Setup**: Docker Compose POCs where the app auto-creates topics on boot and gets restarted; multi-replica deployments where every pod runs the same provisioning code; CI pipelines that re-run topic setup against a persistent cluster. ## 2\. How to Reproduce It Minimal KRaft cluster: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.0 container_name: kafka ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false" ``` ```bash docker compose up -d ``` Create the topic once, then create it again — the second call fails: ```bash kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1 # Created topic orders. kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1 # Error while executing topic command : Topic 'orders' already exists. ``` The programmatic version (`kafka-clients` 3.9.0): ```java try (Admin admin = Admin.create(Map.of( AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"))) { NewTopic topic = new NewTopic("orders", 3, (short) 1); // First call: succeeds. admin.createTopics(List.of(topic)).all().get(); // Second call: throws ExecutionException -> TopicExistsException. admin.createTopics(List.of(topic)).all().get(); } ``` The race variant needs no second manual step — just start two instances of the same provisioning code at once (two pods, or two JVMs), each holding a `NewTopic("orders", ...)`. Both send `CreateTopics` before either sees the other's result. One wins; the other's `KafkaFuture` completes with `TopicExistsException`. In CI this looks flaky: green on a fresh cluster, red on a re-run against a persistent one. ## 3\. Why It Happens — Surface Level Topic creation is not idempotent by default. `CreateTopics` means "create a brand-new topic," not "ensure this topic exists." If a topic with that name is already present in cluster metadata, the broker rejects the request with `TOPIC_ALREADY_EXISTS`. That is the entire surface cause. Your code — or Spring, or the CLI — then propagates that rejection as a fatal exception because it treated a "create" as "create or ignore." The three concrete triggers: 1. **Re-running create logic** against a cluster that already has the topic (restarts, CI replays, dev loops). 2. **Two instances racing** to create the same topic on startup. 3. **A partition/RF mismatch** — you think you are changing the topic, but `CreateTopics` can't modify an existing topic; it only ever rejects it. ## 4\. Why It Happens — Under the Hood In KRaft mode, `AdminClient.createTopics` sends a `CreateTopics` request to the **active controller** (the `QuorumController`). The controller is the single writer of cluster metadata. It replays the request against the current metadata image: for each topic in the batch it checks whether a `TopicRecord` with that name already exists in the `__cluster_metadata` log. If it does, that topic's result in the response is stamped with error code 36 and no `TopicRecord` is written. The controller processes the batch atomically per-topic, so in a multi-topic request the topics that don't exist are still created; only the colliding ones fail. That is why `createTopics(...).values()` gives you a **per-topic** `KafkaFuture` — you can have partial success. This design is what makes the race deterministic rather than random. Metadata changes are serialized through the controller's single-threaded event loop. When two instances fire `CreateTopics` for `orders`, the controller orders them: the first mutation appends a `TopicRecord` and the topic now exists in the metadata image; the second is replayed against that updated image and sees the topic, so it returns 36\. There is no lock to contend on and no corruption — one create, one rejection, every time. (Under ZooKeeper the same outcome came from a znode existence check under `/brokers/topics`; the API contract is identical.) Two subtleties that trip people up: - **`CreateTopics` never modifies.** There is no "upsert." To add partitions you call `createPartitions`; to change configs you call `incrementalAlterConfigs`. Handing `CreateTopics` a `NewTopic` with a different partition count than the live topic does **not** reshape it — it just returns `TopicExistsException`. The mismatch is invisible unless you inspect the live topic yourself. - **`validateOnly` and defaults.** `CreateTopicsOptions.validateOnly(true)` runs the checks without writing — and a `validateOnly` create of an existing topic still returns `TOPIC_ALREADY_EXISTS`. Since KIP-464 (Kafka 2.4) you can pass `-1` for partitions and replication factor to accept broker defaults (`num.partitions`, `default.replication.factor`), but that only affects creation, not the existence check. Spring's `KafkaAdmin` deserves its own note because its behavior is version-dependent. On startup it collects all `NewTopic` beans and reconciles them: it describes existing topics, **creates only the missing ones**, and — if a bean declares *more* partitions than the live topic — issues `createPartitions` to grow it. So in the common case Spring **does not** throw `TopicExistsException`; it swallows the existing-topic case for you. You hit the exception in Spring when reconciliation itself fails — e.g. a bean asks for *fewer* partitions than exist (illegal, surfaces as `InvalidPartitionsException`), or you bypass `KafkaAdmin` and call `AdminClient` directly, or `fatalIfBrokerNotAvailable` interacts with a create you issued by hand. If a bare `TopicExistsException` is taking down your Spring context, something is calling `createTopics` outside the `KafkaAdmin` reconciliation path. ## 5\. The Fix The fix is to make provisioning idempotent: treat `TopicExistsException` as success, not failure. Do **not** retry it, and do **not** let it abort startup. **Before** — fatal on a benign condition: ```java NewTopic topic = new NewTopic("orders", 3, (short) 1); admin.createTopics(List.of(topic)).all().get(); // throws if topic exists ``` **After** — catch and treat as idempotent: ```java NewTopic topic = new NewTopic("orders", 3, (short) 1); try { admin.createTopics(List.of(topic)).all().get(); log.info("Created topic {}", topic.name()); } catch (ExecutionException e) { if (e.getCause() instanceof TopicExistsException) { log.info("Topic {} already exists — skipping", topic.name()); } else { throw e; // real failure: rethrow } } ``` For multi-topic batches, unwrap **per topic** so one existing topic doesn't hide real failures on the others: ```java CreateTopicsResult result = admin.createTopics(List.of( new NewTopic("orders", 3, (short) 1), new NewTopic("payments", 3, (short) 1))); result.values().forEach((name, future) -> { try { future.get(); log.info("Created {}", name); } catch (ExecutionException e) { if (e.getCause() instanceof TopicExistsException) { log.info("{} already exists — skipping", name); } else { throw new RuntimeException("Failed to create " + name, e.getCause()); } } catch (InterruptedException e) { Thread.currentThread().interrupt(); throw new RuntimeException(e); } }); ``` On the CLI, use `--if-not-exists` to make create idempotent: ```bash kafka-topics.sh --bootstrap-server localhost:9092 --create \ --topic orders --partitions 3 --replication-factor 1 --if-not-exists ``` **If the real problem is a partition-count change**, stop reaching for `createTopics`. Grow partitions explicitly (this only ever *increases* — you cannot shrink): ```java admin.createPartitions(Map.of( "orders", NewPartitions.increaseTo(6))).all().get(); ``` Which fix, when: - **POC / single instance**: the catch-and-log pattern above is enough. - **Multi-instance startup**: same catch, but now it is load-bearing — every replica runs it and all but one will legitimately see `TopicExistsException`. That is expected, not a bug. - **You need a different shape**: describe first, then `createPartitions` / `incrementalAlterConfigs`. Never assume `createTopics` reshapes anything. ## 6\. Best Practices & The Better Design The class of bug here is "application code owns topic lifecycle." App-side auto-provisioning is fine for POCs and convenient in dev, but it forces every instance to race on startup and makes topic shape drift silently. **Better design 1 — let Spring's `KafkaAdmin` own it.** Declare topics as beans and let reconciliation handle existence and growth: ```java @Configuration public class KafkaTopicConfig { @Bean public KafkaAdmin.NewTopics appTopics() { return new KafkaAdmin.NewTopics( TopicBuilder.name("orders").partitions(3).replicas(1).build(), TopicBuilder.name("payments").partitions(3).replicas(1) .config(TopicConfig.RETENTION_MS_CONFIG, "604800000") .build() ); } } ``` `KafkaAdmin` creates missing topics, grows partitions where a bean asks for more, and — critically — does **not** throw when the topics already exist. This is the idempotent path you'd otherwise hand-roll. **Better design 2 — topics as code, out of the app entirely.** In any real environment, provision topics with a declarative tool that reconciles desired vs actual state: Strimzi `KafkaTopic` CRDs on Kubernetes, Terraform's Kafka provider, or a gated setup job in CI. The application then assumes its topics exist and never calls `createTopics`. This removes the startup race, kills the "which instance creates it" question, and makes partition/RF/retention reviewable in a pull request: ```yaml # Strimzi KafkaTopic apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic metadata: name: orders labels: strimzi.io/cluster: my-cluster spec: partitions: 3 replicas: 3 config: retention.ms: "604800000" min.insync.replicas: "2" ``` **Better design 3 — turn off broker auto-creation.** Set `auto.create.topics.enable=false` on the cluster. This isn't about `TopicExistsException` directly, but it forces topics to be created deliberately (through one of the paths above) instead of appearing implicitly on first produce/consume — which is what leads to duplicated, racing create logic in the first place. ## 7\. How to Prevent It Long-Term - **Never treat `createTopics` as create-or-update.** Bake the `TopicExistsException`\-is-success rule into your provisioning helper and reuse it everywhere. - **Make provisioning declarative.** Strimzi / Terraform / a single CI job as the one owner of topic state. App instances don't create topics. - **Guard partition/config drift in CI.** A pipeline check that describes live topics and diffs them against the declared spec catches "someone changed partitions in a `NewTopic` bean and it silently did nothing." - **Alert on the wrong signals.** A steady trickle of `TOPIC_ALREADY_EXISTS` in controller logs during rollouts is normal (racing replicas). A *spike* outside deploys, or `InvalidPartitionsException`, means someone is trying to reshape a topic through the create path — that's the real alertable event. - **Test the race.** A Testcontainers test that starts two `AdminClient`s creating the same topic concurrently and asserts exactly one success + one `TopicExistsException` locks in the idempotent handling so a refactor can't reintroduce the crash. ## 8\. Key Takeaways - `TopicExistsException` (code 36, `TOPIC_ALREADY_EXISTS`) is **non-retriable** and usually **benign** — `createTopics` means "create new," not "ensure exists." - Unwrap it from `ExecutionException` **per topic** and treat it as success; only rethrow the causes that aren't `TopicExistsException`. - The KRaft controller serializes creates through a single-writer event loop, so the multi-instance race is deterministic: one create, one rejection — not corruption. - `createTopics` **never modifies** a topic. Use `createPartitions` to grow and `incrementalAlterConfigs` to change configs; a partition mismatch shows up as `TopicExistsException`, not a reshape. - Best design: don't provision topics from app code. Use Spring `KafkaAdmin` reconciliation for dev, and Strimzi/Terraform (topics as code) for anything real.