Kafka NotControllerException: Why It Happens and How to Fix It

You ran a createTopics, a --delete, a partition reassignment, or an ACL change, and got this:

org.apache.kafka.common.errors.NotControllerException: This is not the correct controller for this cluster.

Or, wrapped, out of an AdminClient future:

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotControllerException:
    This is not the correct controller for this cluster.
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)

This is NOT_CONTROLLER, protocol error code 41. It means the admin request reached a broker that is not the cluster's active controller, and cluster-metadata mutations (create/delete topics, create partitions, reassign, ZK-era ACL/config writes) can only be served by the controller. It is almost never a "your cluster is broken" error — it's a routing-during-election error, and the class hierarchy tells you exactly how to treat it.

1. The Error

The exact class and message:

org.apache.kafka.common.errors.NotControllerException: This is not the correct controller for this cluster.

Critically, the class is retriable:

// org.apache.kafka.common.errors.NotControllerException
public class NotControllerException extends RetriableException { ... }

extends RetriableException (not bare ApiException) is the whole story. Kafka is telling you: try again, this can succeed on retry — the target moved, it wasn't rejected.

Where you hit it:

• kafka-topics.sh --create/--delete, kafka-reassign-partitions.sh, kafka-acls.sh, kafka-configs.sh (all AdminClient under the hood since 2.2).
• Java/Spring AdminClient / KafkaAdmin: createTopics, deleteTopics, createPartitions, createAcls, alterConfigs.
• Broker logs during a controller failover or rolling restart.

Versions and setup where it typically bites: multi-broker clusters mid-election — a rolling restart, a controller crash, a ZK-to-KRaft migration window, or CLI/tools pointed at a broker whose cached controller identity is stale. It shows up far more on older/ZK-era clusters; on modern KRaft clusters (Kafka 3.x/4.0) broker-side forwarding (KIP-590) hides most of it, but it still surfaces during KRaft controller-quorum leadership changes and from tooling that talks directly to a controller listener.

2. How to Reproduce It (step-by-step)

You need more than one broker and a controller that moves. A KRaft cluster with dedicated controllers reproduces it cleanly. Minimal docker-compose.yml with a 3-node controller quorum:

# docker-compose.yml — Kafka 3.9 KRaft, 3 controllers + 1 broker
services:
  controller-1: &ctrl
    image: apache/kafka:3.9.0
    environment: &ctrlenv
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: CONTROLLER://:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      CLUSTER_ID: "kraft-notcontroller-demo-000000"
  controller-2:
    <<: *ctrl
    environment: { <<: *ctrlenv, KAFKA_NODE_ID: 2 }
  controller-3:
    <<: *ctrl
    environment: { <<: *ctrlenv, KAFKA_NODE_ID: 3 }
  broker-1:
    image: apache/kafka:3.9.0
    ports: ["29092:29092"]
    environment:
      KAFKA_NODE_ID: 10
      KAFKA_PROCESS_ROLES: broker
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: PLAINTEXT://:9092,EXTERNAL://:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker-1:9092,EXTERNAL://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      CLUSTER_ID: "kraft-notcontroller-demo-000000"

docker compose up -d

# Find the active controller (ActiveController is the Raft leader of the quorum)
docker compose exec controller-1 \
  /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-controller controller-1:9093 describe --status

Now force leadership to move while you hammer admin operations. In one shell, loop createTopics; in another, kill the current leader:

# shell A — repeatedly create/delete topics via the broker
while true; do
  docker compose exec broker-1 /opt/kafka/bin/kafka-topics.sh \
    --bootstrap-server broker-1:9092 --create --if-not-exists \
    --topic churn-$RANDOM --partitions 3 --replication-factor 1
done

# shell B — kill the active controller to trigger a Raft election
docker compose kill controller-1     # (whichever is leader)

In the election window you'll see the CLI spit NotControllerException before the AdminClient's own retry catches up, and the broker log will show the forwarded request bouncing off a stale controller target. On a ZK-era cluster the classic reproduction is simpler: point kafka-topics.sh --create (or a direct-to-controller tool) at a broker mid-election, or run it against a broker whose cached controllerId hasn't caught up after the /controller znode changed hands.

Environment-specific triggers: rolling restarts (each bounce of the controller node moves leadership), ZK-to-KRaft migration (brokers in pre-migration mode reject writes with NOT_CONTROLLER), and tools/clients that bootstrap against a single broker that happens to be lagging on metadata.

3. Why It Happens — Surface Level

Exactly one node in the cluster is the active controller at any moment. It is the only node allowed to mutate cluster metadata: creating and deleting topics, adding partitions, moving replicas, and (ZK-era) writing ACLs and dynamic configs. Every other broker is a follower for metadata purposes.

An admin request is controller-bound. The client (or a forwarding broker) has to deliver it to whoever is currently the controller. During an election — or whenever a node's cached "who is the controller" is stale — the request lands on a node that was the controller a moment ago, or never was. That node answers NOT_CONTROLLER. Because the exception is retriable, the correct response is to refresh the controller identity and send it again, which is exactly what the modern AdminClient does for you.

4. Why It Happens — Under the Hood

Two eras, one symptom.

ZK era. The controller is elected by racing to create the /controller ephemeral znode in ZooKeeper. The winner owns metadata writes and pushes UpdateMetadata requests to all brokers, including the current controllerId. Each broker caches that in its MetadataCache. When the controller dies, its znode disappears, another broker wins the race, and there is a propagation window where some brokers still advertise the old controllerId. The AdminClient's ControllerNodeProvider routes controller-bound calls to whichever broker the last Metadata response named as controller. Point it at a stale value and you hit the wrong broker → NOT_CONTROLLER.

KIP-590 (Kafka 2.8+). Brokers gained the ability to forward ZK-mutation admin requests to the active controller using Envelope requests. The client no longer has to find the controller itself — it sends the admin request to any broker, and that broker forwards it. This is why NotControllerException largely vanished from client-facing logs on modern clusters: the forwarding broker absorbs the routing and retries internally. You still see it when the broker forwards to a controller that just stopped being the controller.

KRaft (Kafka 3.x, mandatory in 4.0). There is no ZK. Cluster metadata is a replicated log managed by a Raft quorum of controller nodes; the QuorumController running on the Raft leader is the active controller. Metadata writes are appended to the __cluster_metadata log and committed by the quorum. Brokers forward all metadata mutations to the active controller. When Raft leadership moves — leader crash, kafka-metadata-quorum.sh reassignment, a controller rolling restart — the node that was leader rejects appends with NOT_CONTROLLER until the new leader is established and its identity propagates. The ActiveControllerCount you scrape from any single node is the local view; across the cluster it must sum to exactly 1.

The retry contract makes this benign in steady state. On NOT_CONTROLLER, AdminClient doesn't just re-send to the same node — it invalidates its cached controller, forces a metadata refresh, re-runs the ControllerNodeProvider to discover the new controller, and re-dispatches, all inside the configured retries / retry.backoff.ms / default.api.timeout.ms budget. It is categorized internally as "refresh metadata and re-run the node provider," not "sleep and retry the same target." Elections resolve in milliseconds to a few seconds; a sane retry budget swallows the whole window.

5. The Fix

The fix is almost never "restart the cluster." It's "let the retriable error retry, with a budget wide enough to outlast an election." The common bug is calling an admin op with a zero or default-thin budget and treating the first exception as fatal.

Before — one shot, no budget, fatal on first election blip:

// BEFORE: brittle — throws on any controller move
Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");

try (Admin admin = Admin.create(props)) {
    admin.createTopics(List.of(new NewTopic("orders", 6, (short) 3)))
         .all()
         .get();   // NotControllerException surfaces here, uncaught
}

After — give retries and the overall API timeout room to cross an election, and treat the op as idempotent:

// AFTER: survives a controller election window
Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
props.put(AdminClientConfig.RETRIES_CONFIG, Integer.MAX_VALUE);        // time-bounded, not count-bounded
props.put(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG, 200);
props.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, 30_000);        // per attempt
props.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120_000);   // whole operation

try (Admin admin = Admin.create(props)) {
    CreateTopicsResult result = admin.createTopics(
        List.of(new NewTopic("orders", 6, (short) 3)),
        new CreateTopicsOptions().timeoutMs(120_000));
    try {
        result.all().get();
    } catch (ExecutionException e) {
        if (e.getCause() instanceof TopicExistsException) {
            // idempotent: a retried create that already succeeded is fine
        } else {
            throw e;
        }
    }
}

Two things matter here. default.api.timeout.ms (default 60000) bounds the entire operation across retries — it must exceed your worst-case election time, so give it headroom. And because a retry can re-run a createTopics that already committed on the previous attempt, handle TopicExistsException as success. That second point catches the historical AdminClient infinite-retry footgun (KAFKA-7113): a retried create hitting TopicExistsException after the first attempt actually landed.

For the CLI, the fix is even simpler: re-run it. kafka-topics.sh uses AdminClient and will usually ride through the election on its own; a NotControllerException printed once during a rolling restart is not a cluster problem, it's the election resolving. Do not reach for --zookeeper (gone since 3.0) or bounce brokers to "clear" it.

If you're on a cluster older than 2.8 without forwarding, prefer bootstrapping the AdminClient against multiple brokers so a single stale node can't pin your controller lookup, and keep clients current — pre-2.8 clients don't benefit from broker-side forwarding.

6. Best Practices & The Better Design

Stop treating controller elections as errors and start treating them as normal cluster events your tooling must tolerate.

Provision topics and ACLs as code through an operator that already implements retry-on-NOT_CONTROLLER: Strimzi KafkaTopic/KafkaUser CRDs, Terraform's Kafka provider, or JulieOps. These reconcile continuously, so an election mid-apply is retried automatically instead of failing a pipeline. Hand-run kafka-topics.sh in a deploy script is the setup most likely to surface the raw exception.

On KRaft, run dedicated controller nodes separate from brokers. When metadata leadership lives on nodes that aren't also serving produce/fetch traffic, ordinary broker restarts don't move the controller at all, and you eliminate most of the elections that produce this error. Combine that with ISR-gated, one-at-a-time rolling restarts and restart controller-quorum members deliberately (not all at once).

The "right way" for a service that self-provisions its topics at startup:

// Idempotent, election-tolerant topic provisioning at boot
Properties p = new Properties();
p.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap);       // several brokers
p.put(AdminClientConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
p.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120_000);

try (Admin admin = Admin.create(p)) {
    var topics = List.of(new NewTopic("orders", 6, (short) 3));
    admin.createTopics(topics).values().forEach((name, fut) ->
        fut.whenComplete((v, ex) -> {
            if (ex != null && !(ex instanceof TopicExistsException))
                throw new CompletionException(ex);   // real failure
            // else: created, or already existed — both fine
        }));
}

In Spring, let KafkaAdmin + NewTopic beans do this — its reconciliation already swallows the already-exists case and retries transient admin failures, so a controller move during context startup doesn't fail your boot.

7. How to Prevent It Long-Term

Monitor the controller, not the exception. The single most important metric is kafka.controller:type=KafkaController,name=ActiveControllerCount. Scrape it from every node and sum it — the total must equal exactly 1. A sum of 0 means no controller (real outage; a storm of NotControllerException that never resolves). A sum of 2+ means split brain. Alert on both.

Track election churn: ControllerEventManager queue time, KRaft kafka.controller metrics for leadership changes, and LeaderElectionRateAndTimeMs. A steady trickle of NotControllerException during deploys is normal; a sustained stream outside deploy windows means the controller is flapping — chase the cause (GC pauses on the controller node, disk stalls on the metadata log, network partitions to the quorum), don't widen retries to paper over it.

Config conventions for the team: set explicit retries / default.api.timeout.ms on every AdminClient (never rely on a thin default in a CI job), always classify TopicExistsException as success in create paths, and keep bootstrap.servers pointed at multiple brokers. In CI, run a chaos test that kills the active controller mid-createTopics and asserts the operation still completes — that's the test that catches a missing retry budget before production does.

8. Key Takeaways / Learnings

• NotControllerException (code 41) means an admin request hit a node that isn't the active controller. It extends RetriableException — the fix is retry with an adequate budget, not restarting anything.
• AdminClient handles it by refreshing metadata and re-routing to the new controller inside retries / default.api.timeout.ms. Give default.api.timeout.ms room (≥120s) to cross an election.
• KIP-590 broker-side forwarding (2.8+) hides most of it; in KRaft it appears during controller-quorum leadership changes. Keep clients current.
• Make create paths idempotent: treat TopicExistsException after a retry as success (avoids the KAFKA-7113 retry trap).
• Provision topics/ACLs as code (Strimzi/Terraform), run dedicated KRaft controllers, and alert on sum(ActiveControllerCount) != 1 — that's the real signal, not the client exception.