> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka NotControllerException: Why It Happens and How to Fix It - URL: https://www.ggorantala.dev/kafka-notcontrollerexception-fix/ - Published: 2026-01-14T11:16:00.000Z - Updated: 2026-08-29T08:27:05.000Z - Description: Kafka throws NotControllerException (code 41) when an admin request hits a broker that isn't the active controller. Here's the mechanism and the real fix. - Author: Gopi Gorantala - Tags: kafka-errors, NotControllerException, kafka-admin, kraft, kafka-controller, Java, event-streaming You ran a `createTopics`, a `--delete`, a partition reassignment, or an ACL change, and got this: ```text org.apache.kafka.common.errors.NotControllerException: This is not the correct controller for this cluster. ``` Or, wrapped, out of an `AdminClient` future: ```text java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotControllerException: This is not the correct controller for this cluster. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ``` This is `NOT_CONTROLLER`, protocol error **code 41**. It means the admin request reached a broker that is *not* the cluster's active controller, and cluster-metadata mutations (create/delete topics, create partitions, reassign, ZK-era ACL/config writes) can only be served by the controller. It is almost never a "your cluster is broken" error — it's a routing-during-election error, and the class hierarchy tells you exactly how to treat it. ## 1\. The Error The exact class and message: ```text org.apache.kafka.common.errors.NotControllerException: This is not the correct controller for this cluster. ``` Critically, the class is **retriable**: ```java // org.apache.kafka.common.errors.NotControllerException public class NotControllerException extends RetriableException { ... } ``` `extends RetriableException` (not bare `ApiException`) is the whole story. Kafka is telling you: *try again, this can succeed on retry* — the target moved, it wasn't rejected. Where you hit it: - `kafka-topics.sh --create/--delete`, `kafka-reassign-partitions.sh`, `kafka-acls.sh`, `kafka-configs.sh` (all `AdminClient` under the hood since 2.2). - Java/Spring `AdminClient` / `KafkaAdmin`: `createTopics`, `deleteTopics`, `createPartitions`, `createAcls`, `alterConfigs`. - Broker logs during a controller failover or rolling restart. Versions and setup where it typically bites: multi-broker clusters mid-election — a rolling restart, a controller crash, a ZK-to-KRaft migration window, or CLI/tools pointed at a broker whose cached controller identity is stale. It shows up far more on older/ZK-era clusters; on modern KRaft clusters (Kafka 3.x/4.0) broker-side forwarding (KIP-590) hides most of it, but it still surfaces during KRaft controller-quorum leadership changes and from tooling that talks directly to a controller listener. ## 2\. How to Reproduce It (step-by-step) You need more than one broker and a controller that moves. A KRaft cluster with dedicated controllers reproduces it cleanly. Minimal `docker-compose.yml` with a 3-node controller quorum: ```yaml # docker-compose.yml — Kafka 3.9 KRaft, 3 controllers + 1 broker services: controller-1: &ctrl image: apache/kafka:3.9.0 environment: &ctrlenv KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENERS: CONTROLLER://:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT CLUSTER_ID: "kraft-notcontroller-demo-000000" controller-2: <<: *ctrl environment: { <<: *ctrlenv, KAFKA_NODE_ID: 2 } controller-3: <<: *ctrl environment: { <<: *ctrlenv, KAFKA_NODE_ID: 3 } broker-1: image: apache/kafka:3.9.0 ports: ["29092:29092"] environment: KAFKA_NODE_ID: 10 KAFKA_PROCESS_ROLES: broker KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENERS: PLAINTEXT://:9092,EXTERNAL://:29092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker-1:9092,EXTERNAL://localhost:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT CLUSTER_ID: "kraft-notcontroller-demo-000000" ``` ```bash docker compose up -d # Find the active controller (ActiveController is the Raft leader of the quorum) docker compose exec controller-1 \ /opt/kafka/bin/kafka-metadata-quorum.sh \ --bootstrap-controller controller-1:9093 describe --status ``` Now force leadership to move while you hammer admin operations. In one shell, loop `createTopics`; in another, kill the current leader: ```bash # shell A — repeatedly create/delete topics via the broker while true; do docker compose exec broker-1 /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server broker-1:9092 --create --if-not-exists \ --topic churn-$RANDOM --partitions 3 --replication-factor 1 done # shell B — kill the active controller to trigger a Raft election docker compose kill controller-1 # (whichever is leader) ``` In the election window you'll see the CLI spit `NotControllerException` before the AdminClient's own retry catches up, and the broker log will show the forwarded request bouncing off a stale controller target. On a ZK-era cluster the classic reproduction is simpler: point `kafka-topics.sh --create` (or a direct-to-controller tool) at a broker mid-election, or run it against a broker whose cached `controllerId` hasn't caught up after the `/controller` znode changed hands. Environment-specific triggers: rolling restarts (each bounce of the controller node moves leadership), ZK-to-KRaft migration (brokers in pre-migration mode reject writes with `NOT_CONTROLLER`), and tools/clients that bootstrap against a single broker that happens to be lagging on metadata. ## 3\. Why It Happens — Surface Level Exactly one node in the cluster is the active controller at any moment. It is the only node allowed to mutate cluster metadata: creating and deleting topics, adding partitions, moving replicas, and (ZK-era) writing ACLs and dynamic configs. Every other broker is a follower for metadata purposes. An admin request is *controller-bound*. The client (or a forwarding broker) has to deliver it to whoever is currently the controller. During an election — or whenever a node's cached "who is the controller" is stale — the request lands on a node that was the controller a moment ago, or never was. That node answers `NOT_CONTROLLER`. Because the exception is retriable, the correct response is to refresh the controller identity and send it again, which is exactly what the modern `AdminClient` does for you. ## 4\. Why It Happens — Under the Hood Two eras, one symptom. **ZK era.** The controller is elected by racing to create the `/controller` ephemeral znode in ZooKeeper. The winner owns metadata writes and pushes `UpdateMetadata` requests to all brokers, including the current `controllerId`. Each broker caches that in its `MetadataCache`. When the controller dies, its znode disappears, another broker wins the race, and there is a propagation window where some brokers still advertise the *old* `controllerId`. The `AdminClient`'s `ControllerNodeProvider` routes controller-bound calls to whichever broker the last `Metadata` response named as controller. Point it at a stale value and you hit the wrong broker → `NOT_CONTROLLER`. **KIP-590 (Kafka 2.8+).** Brokers gained the ability to *forward* ZK-mutation admin requests to the active controller using `Envelope` requests. The client no longer has to find the controller itself — it sends the admin request to any broker, and that broker forwards it. This is why `NotControllerException` largely vanished from client-facing logs on modern clusters: the forwarding broker absorbs the routing and retries internally. You still see it when the broker forwards to a controller that *just* stopped being the controller. **KRaft (Kafka 3.x, mandatory in 4.0).** There is no ZK. Cluster metadata is a replicated log managed by a Raft quorum of controller nodes; the `QuorumController` running on the Raft *leader* is the active controller. Metadata writes are appended to the `__cluster_metadata` log and committed by the quorum. Brokers forward all metadata mutations to the active controller. When Raft leadership moves — leader crash, `kafka-metadata-quorum.sh` reassignment, a controller rolling restart — the node that *was* leader rejects appends with `NOT_CONTROLLER` until the new leader is established and its identity propagates. The `ActiveControllerCount` you scrape from any single node is the local view; across the cluster it must sum to exactly 1. The retry contract makes this benign in steady state. On `NOT_CONTROLLER`, `AdminClient` doesn't just re-send to the same node — it invalidates its cached controller, forces a metadata refresh, re-runs the `ControllerNodeProvider` to discover the *new* controller, and re-dispatches, all inside the configured `retries` / `retry.backoff.ms` / `default.api.timeout.ms` budget. It is categorized internally as "refresh metadata and re-run the node provider," not "sleep and retry the same target." Elections resolve in milliseconds to a few seconds; a sane retry budget swallows the whole window. ## 5\. The Fix The fix is almost never "restart the cluster." It's "let the retriable error retry, with a budget wide enough to outlast an election." The common bug is calling an admin op with a zero or default-thin budget and treating the first exception as fatal. Before — one shot, no budget, fatal on first election blip: ```java // BEFORE: brittle — throws on any controller move Properties props = new Properties(); props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092"); try (Admin admin = Admin.create(props)) { admin.createTopics(List.of(new NewTopic("orders", 6, (short) 3))) .all() .get(); // NotControllerException surfaces here, uncaught } ``` After — give retries and the overall API timeout room to cross an election, and treat the op as idempotent: ```java // AFTER: survives a controller election window Properties props = new Properties(); props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092"); props.put(AdminClientConfig.RETRIES_CONFIG, Integer.MAX_VALUE); // time-bounded, not count-bounded props.put(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG, 200); props.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, 30_000); // per attempt props.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120_000); // whole operation try (Admin admin = Admin.create(props)) { CreateTopicsResult result = admin.createTopics( List.of(new NewTopic("orders", 6, (short) 3)), new CreateTopicsOptions().timeoutMs(120_000)); try { result.all().get(); } catch (ExecutionException e) { if (e.getCause() instanceof TopicExistsException) { // idempotent: a retried create that already succeeded is fine } else { throw e; } } } ``` Two things matter here. `default.api.timeout.ms` (default 60000) bounds the *entire* operation across retries — it must exceed your worst-case election time, so give it headroom. And because a retry can re-run a `createTopics` that already committed on the previous attempt, handle `TopicExistsException` as success. That second point catches the historical `AdminClient` infinite-retry footgun (KAFKA-7113): a retried create hitting `TopicExistsException` after the first attempt actually landed. For the CLI, the fix is even simpler: re-run it. `kafka-topics.sh` uses `AdminClient` and will usually ride through the election on its own; a `NotControllerException` printed once during a rolling restart is not a cluster problem, it's the election resolving. Do **not** reach for `--zookeeper` (gone since 3.0) or bounce brokers to "clear" it. If you're on a cluster older than 2.8 without forwarding, prefer bootstrapping the `AdminClient` against multiple brokers so a single stale node can't pin your controller lookup, and keep clients current — pre-2.8 clients don't benefit from broker-side forwarding. ## 6\. Best Practices & The Better Design Stop treating controller elections as errors and start treating them as normal cluster events your tooling must tolerate. Provision topics and ACLs *as code* through an operator that already implements retry-on-`NOT_CONTROLLER`: Strimzi `KafkaTopic`/`KafkaUser` CRDs, Terraform's Kafka provider, or JulieOps. These reconcile continuously, so an election mid-apply is retried automatically instead of failing a pipeline. Hand-run `kafka-topics.sh` in a deploy script is the setup most likely to surface the raw exception. On KRaft, run **dedicated controller nodes** separate from brokers. When metadata leadership lives on nodes that aren't also serving produce/fetch traffic, ordinary broker restarts don't move the controller at all, and you eliminate most of the elections that produce this error. Combine that with ISR-gated, one-at-a-time rolling restarts and restart controller-quorum members deliberately (not all at once). The "right way" for a service that self-provisions its topics at startup: ```java // Idempotent, election-tolerant topic provisioning at boot Properties p = new Properties(); p.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap); // several brokers p.put(AdminClientConfig.RETRIES_CONFIG, Integer.MAX_VALUE); p.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120_000); try (Admin admin = Admin.create(p)) { var topics = List.of(new NewTopic("orders", 6, (short) 3)); admin.createTopics(topics).values().forEach((name, fut) -> fut.whenComplete((v, ex) -> { if (ex != null && !(ex instanceof TopicExistsException)) throw new CompletionException(ex); // real failure // else: created, or already existed — both fine })); } ``` In Spring, let `KafkaAdmin` \+ `NewTopic` beans do this — its reconciliation already swallows the already-exists case and retries transient admin failures, so a controller move during context startup doesn't fail your boot. ## 7\. How to Prevent It Long-Term Monitor the controller, not the exception. The single most important metric is `kafka.controller:type=KafkaController,name=ActiveControllerCount`. Scrape it from every node and **sum it — the total must equal exactly 1**. A sum of 0 means no controller (real outage; a storm of `NotControllerException` that never resolves). A sum of 2+ means split brain. Alert on both. Track election churn: `ControllerEventManager` queue time, KRaft `kafka.controller` metrics for leadership changes, and `LeaderElectionRateAndTimeMs`. A steady trickle of `NotControllerException` during deploys is normal; a *sustained* stream outside deploy windows means the controller is flapping — chase the cause (GC pauses on the controller node, disk stalls on the metadata log, network partitions to the quorum), don't widen retries to paper over it. Config conventions for the team: set explicit `retries` / `default.api.timeout.ms` on every `AdminClient` (never rely on a thin default in a CI job), always classify `TopicExistsException` as success in create paths, and keep `bootstrap.servers` pointed at multiple brokers. In CI, run a chaos test that kills the active controller mid-`createTopics` and asserts the operation still completes — that's the test that catches a missing retry budget before production does. ## 8\. Key Takeaways / Learnings - `NotControllerException` (code 41) means an admin request hit a node that isn't the active controller. It **extends `RetriableException`** — the fix is retry with an adequate budget, not restarting anything. - `AdminClient` handles it by refreshing metadata and re-routing to the new controller inside `retries` / `default.api.timeout.ms`. Give `default.api.timeout.ms` room (≥120s) to cross an election. - KIP-590 broker-side forwarding (2.8+) hides most of it; in KRaft it appears during controller-quorum leadership changes. Keep clients current. - Make create paths idempotent: treat `TopicExistsException` after a retry as success (avoids the KAFKA-7113 retry trap). - Provision topics/ACLs as code (Strimzi/Terraform), run dedicated KRaft controllers, and alert on `sum(ActiveControllerCount) != 1` — that's the real signal, not the client exception.