Kafka: Timed out waiting for a node assignment — Fix

You ran kafka-topics.sh --create against a cluster that is clearly up, and 30–60 seconds later the command dies with a timeout that names an internal API call. This is not a slow broker. It is the AdminClient telling you it could not find a single reachable broker to route the request to. The distinction matters, because the usual reflex — bumping request.timeout.ms — buys you nothing here.

1. The Error

The exact text, straight from kafka-topics.sh:

Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2026-07-13 09:14:02,113] ERROR org.apache.kafka.common.errors.TimeoutException:
    Timed out waiting for a node assignment. Call: createTopics
 (kafka.admin.TopicCommand$)

The same failure surfaces from any AdminClient call — the Call: suffix just changes:

org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeCluster
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata

And from application code (Spring Boot KafkaAdmin at startup, or a raw AdminClient):

java.util.concurrent.ExecutionException:
    org.apache.kafka.common.errors.TimeoutException:
        Timed out waiting for a node assignment. Call: createTopics

Environment where this bites: Apache Kafka 3.9 / 4.0 (KRaft), kafka-clients 3.x, almost always local Docker Compose or a fresh k8s deployment. The CLI has used AdminClient under the hood since 2.2 (--zookeeper is gone in 3.x), so kafka-topics.sh, kafka-configs.sh, and kafka-consumer-groups.sh all fail the same way.

2. How to Reproduce It

The cleanest reproduction is the classic Docker listener mistake — a broker that advertises an address the host cannot reach.

# docker-compose.yml — deliberately broken advertised listener
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      # BUG: advertises the in-container hostname to every client
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
# From the host — this hangs, then throws:
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

The TCP connection to localhost:9092 succeeds. The broker answers the Metadata request — and hands back one node whose advertised address is kafka:9092. Your host cannot resolve kafka, so the AdminClient has a cluster with zero usable nodes. Every call waits for a node that never becomes assignable, then times out.

Turn on client logging and the mechanism is unambiguous:

kafka-topics.sh --bootstrap-server localhost:9092 --list \
  --command-config <(echo "log4j.logger.org.apache.kafka.clients=DEBUG")

DEBUG [AdminClient clientId=adminclient-1] Initiating connection to node
      localhost:9092 (id: -1 rack: null)
DEBUG ... Updating cluster metadata to Cluster(nodes = [kafka:9092 (id: 1)])
DEBUG ... Initiating connection to node kafka:9092 (id: 1)
WARN  ... Error connecting to node kafka:9092 (id: 1)
      java.net.UnknownHostException: kafka

Note the two-phase connection: bootstrap node id: -1 succeeds, then the real node id: 1 at the advertised address fails. Same root cause produces the timeout when the port is wrong, the broker is still starting, or a security-protocol mismatch (see below) prevents the second connection from ever completing.

3. Why It Happens — Surface Level

AdminClient does not send your request to whatever socket you put in --bootstrap-server. It uses the bootstrap address only to fetch cluster metadata, then routes each call to a specific broker node from that metadata. "Node assignment" is that routing step. If no node from the metadata is reachable — or if metadata never arrives — the call sits in the pending queue until default.api.timeout.ms expires, and you get Timed out waiting for a node assignment.

The three things that produce it: metadata never returns (can't reach bootstrap at all), or metadata returns broker addresses you can't connect to (the Docker case), or every candidate node is down.

4. Why It Happens — Under the Hood

KafkaAdminClient runs a single I/O thread (AdminClientRunnable) that drives every call through a state machine. A call is created, then must be assigned to a node before it can be sent. Node selection goes through a NodeProvider — for most calls that is LeastLoadedNodeProvider, which asks AdminMetadataManager for the current cluster and picks a connectable node.

Two conditions must both hold for assignment to succeed: the client must have fresh cluster metadata, and at least one node in that metadata must be connectable (or already connected). On each poll of the runnable loop, unassigned calls are retried. When a call can't be assigned, AdminMetadataManager is asked to refresh metadata, which triggers a Metadata request against the bootstrap node.

Here is the trap. The bootstrap connection is a throwaway with synthetic node id -1. The moment metadata comes back, the client switches to the advertised addresses in that metadata and tries to open real connections to them. In the broken-Docker case metadata arrives fine — so the bootstrap leg looks healthy — but every advertised node is unreachable, so LeastLoadedNodeProvider never returns a usable node. The call is retried on every loop iteration until the deadline.

That deadline is default.api.timeout.ms (default 60000 ms), which caps the whole call including all internal retries. request.timeout.ms (default 30000 ms) bounds a single attempt/round-trip. retries defaults to Integer.MAX_VALUE, so retries are effectively bounded by time, not count. This is why raising request.timeout.ms or retries does nothing: the call never gets far enough to send a request that could time out. It is starving at the assignment stage, and only default.api.timeout.ms governs how long it starves before throwing.

A frequently-missed variant: a security-protocol mismatch produces the identical message. Point a plaintext CLI at a SASL_SSL listener and the bootstrap TCP connect succeeds, but the SASL/TLS handshake never completes, so metadata never arrives, no node is assignable, and you get the same timeout — with no obvious "auth failed" line, because the handshake stalls rather than being rejected.

5. The Fix

Fix the address the broker advertises, or fix the address/protocol the client dials. The timeout is a symptom; correcting reachability is the fix.

Docker: advertise a host-reachable listener. Split internal (container-to-container) and external (host-to-container) traffic onto separate listeners.

-      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
-      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+      KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:29092,CONTROLLER://:9093
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:29092
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
-      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT

     ports:
-      - "9092:9092"
+      - "29092:29092"

From the host you now connect to the external listener, whose advertised address (localhost:29092) is reachable:

kafka-topics.sh --bootstrap-server localhost:29092 --create \
  --topic orders --partitions 3 --replication-factor 1

Other containers still use kafka:9092. Nobody is handed an address they cannot reach.

Security mismatch: give the CLI a matching client config. If the listener is SASL_SSL, the CLI must speak it:

# client.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule \
  required username="admin" password="admin-secret";
ssl.truststore.location=/certs/kafka.truststore.jks
ssl.truststore.password=changeit

-kafka-topics.sh --bootstrap-server broker:9093 --list
+kafka-topics.sh --bootstrap-server broker:9093 --list \
+  --command-config client.properties

Broker not ready yet (CI/startup races): if the broker is genuinely still coming up, the honest fix is to wait for readiness before issuing admin calls, not to widen the timeout. But if you must tolerate a slow start, raise the call ceiling — this is the only situation where the timeout knob is the right answer:

-adminProps.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 60000);
+adminProps.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120000);

6. Best Practices & The Better Design

Treat advertised.listeners as a network contract: every caller must be handed an address it can actually open a socket to, from where it runs. Get that right and this class of error disappears. The dual INTERNAL/EXTERNAL pattern above is the canonical shape — in Kubernetes it generalizes to per-broker stable DNS or NodePort, never a shared Service VIP, because clients need to reach specific brokers by their advertised names.

For application code, verify reachability explicitly at startup instead of letting the first createTopics be your connectivity probe:

try (Admin admin = Admin.create(adminProps)) {
    // describeCluster is the cheapest "can I reach a node?" call
    DescribeClusterResult cluster = admin.describeCluster(
        new DescribeClusterOptions().timeoutMs(10_000));
    Collection<Node> nodes = cluster.nodes().get();
    log.info("Connected to cluster {} with {} broker(s)",
        cluster.clusterId().get(), nodes.size());

    admin.createTopics(List.of(
        new NewTopic("orders", 3, (short) 1)
    )).all().get();
} catch (ExecutionException e) {
    if (e.getCause() instanceof TimeoutException) {
        // Not a slow broker — an unreachable one. Log the advertised
        // addresses so the operator sees WHERE it tried to connect.
        log.error("No broker was assignable — check advertised.listeners "
            + "and bootstrap reachability", e.getCause());
    }
    throw e;
}

Failing fast on describeCluster gives you a clear "cannot reach any broker" signal with the node list in the logs, instead of a cryptic per-call timeout deep in your topic-provisioning path.

Better still, do not provision topics from imperative CLI calls at all. Own them as code — Strimzi KafkaTopic CRDs, Terraform, or a Spring KafkaAdmin bean with declared NewTopics — so creation is idempotent and reconciled, and the reachability contract is validated once in your platform config rather than rediscovered on every manual kafka-topics.sh invocation.

7. How to Prevent It Long-Term

Validate the listener setup, not just the broker's health. A liveness check that only confirms the broker process is up will pass while every external client times out. Your smoke test must actually connect an AdminClient from the client's network location and run describeCluster — in CI, run it from a separate container/network namespace, not from inside the broker container where kafka:9092 happens to resolve.

Lint advertised.listeners in config-as-code: reject an internal-only hostname on an externally-published listener, and reject a listener published to the host without a corresponding host-reachable advertised address. This one check catches the overwhelming majority of Docker and k8s occurrences before they ship.

For running clusters, alert on the client side. There is no single broker metric for "clients can't route to me," so watch for AdminClient/producer/consumer calls timing out at bootstrap: elevated TimeoutException rates from admin tooling, or new consumers stuck with assigned-partitions=0. Keep a short reachability runbook — kafka-broker-api-versions.sh --bootstrap-server <addr> from the affected host prints the advertised address of every broker it discovers, which immediately shows whether metadata is handing back names the caller can't reach.

8. Key Takeaways

• Timed out waiting for a node assignment is a routing failure, not a slow broker. The AdminClient fetched (or failed to fetch) metadata and found no reachable node to send the call to.
• Bootstrap success proves nothing. The client connects to the bootstrap address only to get metadata, then dials the advertised addresses — if those are unreachable (Docker kafka:9092), every call starves and times out.
• request.timeout.ms and retries won't help. The call never reaches the send stage; only default.api.timeout.ms (60s) bounds the starvation. Raising timeouts only masks a genuinely slow startup.
• A plaintext CLI against a SASL_SSL listener throws the same message — the handshake stalls, metadata never lands. Pass --command-config.
• Fix the contract: split INTERNAL/EXTERNAL listeners so every caller gets a host-reachable advertised address, provision topics as code, and probe with describeCluster at startup instead of letting createTopics be your connectivity test.