> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka: Timed out waiting for a node assignment — Fix - URL: https://www.ggorantala.dev/kafka-timed-out-waiting-for-node-assignment-fix/ - Published: 2026-06-28T14:06:00.000Z - Updated: 2026-08-29T08:22:27.000Z - Description: Kafka AdminClient and kafka-topics.sh fail with 'Timed out waiting for a node assignment. Call: createTopics'. Here's why it happens and the exact fix. - Author: Gopi Gorantala - Tags: kafka-errors, TimeoutException, kafka-adminclient, kafka-topics, kafka-docker, Java, event-streaming You ran `kafka-topics.sh --create` against a cluster that is clearly up, and 30–60 seconds later the command dies with a timeout that names an internal API call. This is not a slow broker. It is the `AdminClient` telling you it could not find a single reachable broker to route the request to. The distinction matters, because the usual reflex — bumping `request.timeout.ms` — buys you nothing here. ## 1\. The Error The exact text, straight from `kafka-topics.sh`: ```text Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics [2026-07-13 09:14:02,113] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics (kafka.admin.TopicCommand$) ``` The same failure surfaces from any `AdminClient` call — the `Call:` suffix just changes: ```text org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeCluster org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata ``` And from application code (Spring Boot `KafkaAdmin` at startup, or a raw `AdminClient`): ```text java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics ``` Environment where this bites: Apache Kafka 3.9 / 4.0 (KRaft), `kafka-clients` 3.x, almost always local Docker Compose or a fresh k8s deployment. The CLI has used `AdminClient` under the hood since 2.2 (`--zookeeper` is gone in 3.x), so `kafka-topics.sh`, `kafka-configs.sh`, and `kafka-consumer-groups.sh` all fail the same way. ## 2\. How to Reproduce It The cleanest reproduction is the classic Docker listener mistake — a broker that advertises an address the host cannot reach. ```yaml # docker-compose.yml — deliberately broken advertised listener services: kafka: image: apache/kafka:3.9.0 ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 # BUG: advertises the in-container hostname to every client KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ``` ```bash docker compose up -d # From the host — this hangs, then throws: kafka-topics.sh --bootstrap-server localhost:9092 \ --create --topic orders --partitions 3 --replication-factor 1 ``` The TCP connection to `localhost:9092` succeeds. The broker answers the `Metadata` request — and hands back one node whose advertised address is `kafka:9092`. Your host cannot resolve `kafka`, so the `AdminClient` has a cluster with zero *usable* nodes. Every call waits for a node that never becomes assignable, then times out. Turn on client logging and the mechanism is unambiguous: ```bash kafka-topics.sh --bootstrap-server localhost:9092 --list \ --command-config <(echo "log4j.logger.org.apache.kafka.clients=DEBUG") ``` ```text DEBUG [AdminClient clientId=adminclient-1] Initiating connection to node localhost:9092 (id: -1 rack: null) DEBUG ... Updating cluster metadata to Cluster(nodes = [kafka:9092 (id: 1)]) DEBUG ... Initiating connection to node kafka:9092 (id: 1) WARN ... Error connecting to node kafka:9092 (id: 1) java.net.UnknownHostException: kafka ``` Note the two-phase connection: bootstrap node `id: -1` succeeds, then the *real* node `id: 1` at the advertised address fails. Same root cause produces the timeout when the port is wrong, the broker is still starting, or a security-protocol mismatch (see below) prevents the second connection from ever completing. ## 3\. Why It Happens — Surface Level `AdminClient` does not send your request to whatever socket you put in `--bootstrap-server`. It uses the bootstrap address *only* to fetch cluster metadata, then routes each call to a specific broker node from that metadata. "Node assignment" is that routing step. If no node from the metadata is reachable — or if metadata never arrives — the call sits in the pending queue until `default.api.timeout.ms` expires, and you get `Timed out waiting for a node assignment`. The three things that produce it: metadata never returns (can't reach bootstrap at all), or metadata returns broker addresses you can't connect to (the Docker case), or every candidate node is down. ## 4\. Why It Happens — Under the Hood `KafkaAdminClient` runs a single I/O thread (`AdminClientRunnable`) that drives every call through a state machine. A call is created, then must be *assigned to a node* before it can be sent. Node selection goes through a `NodeProvider` — for most calls that is `LeastLoadedNodeProvider`, which asks `AdminMetadataManager` for the current cluster and picks a connectable node. Two conditions must both hold for assignment to succeed: the client must have fresh cluster metadata, and at least one node in that metadata must be connectable (or already connected). On each poll of the runnable loop, unassigned calls are retried. When a call can't be assigned, `AdminMetadataManager` is asked to refresh metadata, which triggers a `Metadata` request against the bootstrap node. Here is the trap. The bootstrap connection is a throwaway with synthetic node id `-1`. The moment metadata comes back, the client switches to the *advertised* addresses in that metadata and tries to open real connections to them. In the broken-Docker case metadata arrives fine — so the bootstrap leg looks healthy — but every advertised node is unreachable, so `LeastLoadedNodeProvider` never returns a usable node. The call is retried on every loop iteration until the deadline. That deadline is `default.api.timeout.ms` (default 60000 ms), which caps the whole call including all internal retries. `request.timeout.ms` (default 30000 ms) bounds a single attempt/round-trip. `retries` defaults to `Integer.MAX_VALUE`, so retries are effectively bounded by time, not count. This is why raising `request.timeout.ms` or `retries` does nothing: the call never gets far enough to send a request that could time out. It is starving at the assignment stage, and only `default.api.timeout.ms` governs how long it starves before throwing. A frequently-missed variant: a security-protocol mismatch produces the identical message. Point a plaintext CLI at a `SASL_SSL` listener and the bootstrap TCP connect succeeds, but the SASL/TLS handshake never completes, so metadata never arrives, no node is assignable, and you get the same timeout — with no obvious "auth failed" line, because the handshake stalls rather than being rejected. ## 5\. The Fix Fix the address the broker advertises, or fix the address/protocol the client dials. The timeout is a symptom; correcting reachability is the fix. **Docker: advertise a host-reachable listener.** Split internal (container-to-container) and external (host-to-container) traffic onto separate listeners. ```diff - KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 - KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 + KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:29092,CONTROLLER://:9093 + KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:29092 + KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT ``` ```diff ports: - - "9092:9092" + - "29092:29092" ``` From the host you now connect to the external listener, whose advertised address (`localhost:29092`) is reachable: ```bash kafka-topics.sh --bootstrap-server localhost:29092 --create \ --topic orders --partitions 3 --replication-factor 1 ``` Other containers still use `kafka:9092`. Nobody is handed an address they cannot reach. **Security mismatch: give the CLI a matching client config.** If the listener is `SASL_SSL`, the CLI must speak it: ```properties # client.properties security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule \ required username="admin" password="admin-secret"; ssl.truststore.location=/certs/kafka.truststore.jks ssl.truststore.password=changeit ``` ```diff -kafka-topics.sh --bootstrap-server broker:9093 --list +kafka-topics.sh --bootstrap-server broker:9093 --list \ + --command-config client.properties ``` **Broker not ready yet (CI/startup races):** if the broker is genuinely still coming up, the honest fix is to wait for readiness before issuing admin calls, not to widen the timeout. But if you must tolerate a slow start, raise the call ceiling — this is the *only* situation where the timeout knob is the right answer: ```diff -adminProps.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 60000); +adminProps.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120000); ``` ## 6\. Best Practices & The Better Design Treat `advertised.listeners` as a network contract: every caller must be handed an address it can actually open a socket to, from where it runs. Get that right and this class of error disappears. The dual INTERNAL/EXTERNAL pattern above is the canonical shape — in Kubernetes it generalizes to per-broker stable DNS or NodePort, never a shared Service VIP, because clients need to reach specific brokers by their advertised names. For application code, verify reachability explicitly at startup instead of letting the first `createTopics` be your connectivity probe: ```java try (Admin admin = Admin.create(adminProps)) { // describeCluster is the cheapest "can I reach a node?" call DescribeClusterResult cluster = admin.describeCluster( new DescribeClusterOptions().timeoutMs(10_000)); Collection nodes = cluster.nodes().get(); log.info("Connected to cluster {} with {} broker(s)", cluster.clusterId().get(), nodes.size()); admin.createTopics(List.of( new NewTopic("orders", 3, (short) 1) )).all().get(); } catch (ExecutionException e) { if (e.getCause() instanceof TimeoutException) { // Not a slow broker — an unreachable one. Log the advertised // addresses so the operator sees WHERE it tried to connect. log.error("No broker was assignable — check advertised.listeners " + "and bootstrap reachability", e.getCause()); } throw e; } ``` Failing fast on `describeCluster` gives you a clear "cannot reach any broker" signal with the node list in the logs, instead of a cryptic per-call timeout deep in your topic-provisioning path. Better still, do not provision topics from imperative CLI calls at all. Own them as code — Strimzi `KafkaTopic` CRDs, Terraform, or a Spring `KafkaAdmin` bean with declared `NewTopic`s — so creation is idempotent and reconciled, and the reachability contract is validated once in your platform config rather than rediscovered on every manual `kafka-topics.sh` invocation. ## 7\. How to Prevent It Long-Term Validate the listener setup, not just the broker's health. A liveness check that only confirms the broker process is up will pass while every external client times out. Your smoke test must actually connect an `AdminClient` from the *client's* network location and run `describeCluster` — in CI, run it from a separate container/network namespace, not from inside the broker container where `kafka:9092` happens to resolve. Lint `advertised.listeners` in config-as-code: reject an internal-only hostname on an externally-published listener, and reject a listener published to the host without a corresponding host-reachable advertised address. This one check catches the overwhelming majority of Docker and k8s occurrences before they ship. For running clusters, alert on the client side. There is no single broker metric for "clients can't route to me," so watch for `AdminClient`/producer/consumer calls timing out at bootstrap: elevated `TimeoutException` rates from admin tooling, or new consumers stuck with `assigned-partitions=0`. Keep a short reachability runbook — `kafka-broker-api-versions.sh --bootstrap-server ` from the affected host prints the advertised address of every broker it discovers, which immediately shows whether metadata is handing back names the caller can't reach. ## 8\. Key Takeaways - **`Timed out waiting for a node assignment` is a routing failure, not a slow broker.** The `AdminClient` fetched (or failed to fetch) metadata and found no reachable node to send the call to. - **Bootstrap success proves nothing.** The client connects to the bootstrap address only to get metadata, then dials the *advertised* addresses — if those are unreachable (Docker `kafka:9092`), every call starves and times out. - **`request.timeout.ms` and `retries` won't help.** The call never reaches the send stage; only `default.api.timeout.ms` (60s) bounds the starvation. Raising timeouts only masks a genuinely slow startup. - **A plaintext CLI against a SASL\_SSL listener throws the same message** — the handshake stalls, metadata never lands. Pass `--command-config`. - **Fix the contract:** split INTERNAL/EXTERNAL listeners so every caller gets a host-reachable advertised address, provision topics as code, and probe with `describeCluster` at startup instead of letting `createTopics` be your connectivity test.