> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka Timeout Before Position for Partition Could Be Determined - URL: https://www.ggorantala.dev/kafka-timeout-position-partition-could-be-determined-fix/ - Published: 2026-06-30T14:09:00.000Z - Updated: 2026-08-29T08:22:11.000Z - Description: Kafka consumer failing with TimeoutException: Timeout of 60000ms expired before the position for partition could be determined? Here's the fix. - Author: Gopi Gorantala - Tags: kafka-errors, TimeoutException, kafka-consumer, kafka-docker, offset-management, Java Your consumer starts, joins the group, gets partitions assigned — and then, 60 seconds later, dies with this: ``` org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition orders-3 could be determined ``` No data consumed. No offset committed. The consumer had partitions but never figured out *where* to start reading them. This is not a networking flake you retry past — it's a metadata/routing failure on the **position-resolution path**, and it will loop forever until you fix the underlying leader or listener problem. This article is specifically about the *position* timeout, which is mechanically distinct from the producer-side "not present in metadata" timeout, the group-coordinator "Marking coordinator dead" loop, and `OffsetOutOfRangeException`. Same word "timeout," completely different code path. ## 1\. The Error The exact message, as it lands in logs and gets pasted into search engines: ``` org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition orders-3 could be determined ``` Framework-wrapped variants you'll also see: ```text # Spring Kafka container thread org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition orders-3 could be determined # From endOffsets()/beginningOffsets() in a lag exporter or admin tool org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the last committed offset for partitions [orders-3] could be determined ``` Environment where it bites: - **Client:** `kafka-clients` 2.0 through 4.0 (classic `KafkaConsumer` and the KIP-848 `AsyncKafkaConsumer` share this contract), Spring Kafka 2.x–3.x, Kafka Connect, Kafka Streams, Beam/Druid Kafka connectors. - **Broker:** any 2.x–4.0 line, KRaft or ZooKeeper. - **Setup:** Docker/K8s local clusters with a misrouted per-partition leader, or any cluster running `RF=1` topics where a broker went down. The `60000ms` in the message is not a coincidence — it's the default of `default.api.timeout.ms` (introduced by KIP-266 in Kafka 2.0). That number is your first clue about which knob is involved. ## 2\. How to Reproduce It The cleanest reproduction is an `RF=1` topic whose leader broker is unreachable from the consumer. Single-broker KRaft compose: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.0 ports: - "29092:29092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:29092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 ``` Create the topic and a consumer group with a committed offset: ```bash docker compose up -d # create a 4-partition topic docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --create --topic orders --partitions 4 --replication-factor 1 # produce a few records and consume once so the group has committed offsets docker compose exec kafka bash -c \ 'seq 10 | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders' docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9092 --topic orders \ --group demo --from-beginning --timeout-ms 5000 ``` Now minimal Java consumer that resolves position explicitly (frameworks do this for you under the hood): ```java // kafka-clients 3.9.0 Properties p = new Properties(); p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092"); p.put(ConsumerConfig.GROUP_ID_CONFIG, "demo"); p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); try (KafkaConsumer c = new KafkaConsumer<>(p)) { c.subscribe(List.of("orders")); c.poll(Duration.ofSeconds(2)); // triggers join + assignment for (TopicPartition tp : c.assignment()) { System.out.println(tp + " @ " + c.position(tp)); // <-- blocks here } } ``` **Trigger the failure.** Kill the only broker that leads `orders`, then run the consumer: ```bash docker compose stop kafka ``` With the leader gone, `position()` (and the internal position resolution inside `poll()`) can never complete its `ListOffsets` round-trip. After `default.api.timeout.ms` \= 60s, it throws. The **more common production trigger** doesn't require killing anything: run the consumer from your host machine while the broker only advertises the internal address `kafka:9092`. Bootstrap to `localhost:29092` succeeds, metadata comes back, but the per-partition **leader** is advertised as `kafka:9092`, which your host can't resolve. Position resolution starves against an unroutable leader — identical error, and nothing is actually "down." ## 3\. Why It Happens — Surface Level Before a consumer can fetch a single byte, it must establish a **position** for every assigned partition: the exact offset to read from next. That position comes from one of two places — the last committed offset (fetched from the group coordinator), or a reset offset derived from `auto.offset.reset` (fetched from the **partition leader** via a `ListOffsets` request). Either way, the client has to talk to a specific broker and get an answer. If that broker can't be reached — because the partition has no leader, the leader is on a downed broker, or the leader's advertised address is unroutable from the client — the request never completes. The consumer keeps retrying inside the blocking call until `default.api.timeout.ms` (60s) elapses, then surfaces `TimeoutException`. The word "position" in the message is the tell: this is the offset-resolution step, not fetching, not committing, not group coordination. ## 4\. Why It Happens — Under the Hood Position resolution runs inside `updateFetchPositions()`, called on every `poll()` and directly by `position()`, `endOffsets()`, `beginningOffsets()`, and `offsetsForTimes()`. It has three sub-steps, each of which can stall: 1. `**refreshCommittedOffsetsIfNeeded**` — an `OffsetFetch` RPC to the **group coordinator** (the broker leading the relevant `__consumer_offsets` partition). If a committed offset exists, it becomes the position. Coordinator unreachable → this stalls (and overlaps the "Marking the coordinator dead" symptom, but here it surfaces as the *position* timeout because that's the caller). 2. `**resetPositionsIfNeeded**` — for partitions with no valid committed offset, a `ListOffsets` RPC to the **partition leader** resolves the earliest/latest offset per `auto.offset.reset`. This is the step that requires the *partition leader* specifically, and it's the one that starves when a leader is offline or unroutable. 3. `**validatePositionsIfNeeded**` — KIP-320 leader-epoch validation via `OffsetsForLeaderEpoch`, also sent to the partition leader, to detect truncation after an unclean leadership change. Another leader-dependent round-trip. The critical detail: `ListOffsets` and `OffsetsForLeaderEpoch` are routed to the **leader of each partition**, resolved from cluster metadata. Bootstrap connectivity proves nothing about leader connectivity — they are independent connection paths to potentially different advertised addresses. That's why "the broker is up and my bootstrap works" and "position times out" are perfectly consistent states. Two clocks govern the wait. `request.timeout.ms` (default 30000ms) bounds a *single* RPC attempt; on failure the client refreshes metadata and retries. `default.api.timeout.ms` (default 60000ms) bounds the *whole* blocking operation across all those retries. Because retries are time-bounded, not count-bounded, the consumer will happily re-send `ListOffsets` to a dead leader for a full 60 seconds before giving up. When a partition is genuinely leaderless (`Leader: none`, empty ISR after an `RF=1` broker loss), metadata refresh returns the same "no leader" answer each cycle and the operation is doomed from the first attempt — it just takes 60s to admit it. ## 5\. The Fix First, **identify which partition and whether it has a leader**. Don't guess: ```bash kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders ``` ```text Topic: orders Partition: 3 Leader: none Replicas: 2 Isr: # offline — no leader Topic: orders Partition: 3 Leader: 2 Replicas: 2 Isr: 2 # leader exists — routing problem ``` - `**Leader: none**` → the partition is offline. Restore the broker hosting replica 2, or reassign leadership. On `RF=1` there is no other replica to elect, so the data-bearing broker *must* come back (or you accept data loss and recreate). This is the single-replica trap. - **`Leader: 2` but consumer still times out** → the leader exists but the client can't route to its advertised address. It's a listener problem, not a broker problem. For the **advertised-listener** case (the most common in Docker/K8s), give every caller a reachable address. Before/after: ```diff # broker config — advertised.listeners is a network contract, not decoration - KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 + KAFKA_LISTENERS: PLAINTEXT://:9092,EXTERNAL://:29092 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:29092 + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT ``` ```diff # host-side consumer bootstraps on the listener whose advertised addr it can reach - p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092"); + p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092"); ``` Verify the leader is actually routable from where the client runs: ```bash kafka-broker-api-versions.sh --bootstrap-server localhost:29092 | head # each broker line prints its advertised host:port — must be reachable from THIS box ``` **Fail fast instead of hanging 60s.** If a downstream call (a lag exporter, a health check) calls `position()` or `endOffsets()`, pass an explicit `Duration` so it errors in seconds, not a minute: ```diff - long end = consumer.endOffsets(partitions).get(tp); + // bound the call; catch and report unreachable leaders promptly + Map end = + consumer.endOffsets(partitions, Duration.ofSeconds(5)); ``` **Raising `default.api.timeout.ms` is not the fix.** It only helps the narrow case of a genuinely slow-but-healthy metadata path (a large cluster mid-election). If the leader is offline or unroutable, a bigger timeout just makes you wait longer for the same failure: ```diff # stopgap ONLY for confirmed slow-metadata clusters — not for offline/unroutable leaders - # default 60000 + p.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120000); ``` ## 6\. Best Practices & The Better Design The whole class of failure disappears when **no partition can be leaderless from a single broker loss** and **every advertised address is reachable by its clients**. Replication that survives one failure: ```bash # never run RF=1 for anything a consumer must read continuously kafka-topics.sh --bootstrap-server broker:9092 --create \ --topic orders --partitions 12 --replication-factor 3 \ --config min.insync.replicas=2 ``` With `RF=3`, losing one broker triggers a sub-second leader election to a surviving in-sync replica; position resolution retries and finds the new leader well within the 60s budget. The transient burst self-heals — exactly what the timeout window is *for*. `RF=1` gives it nothing to fall back to. Listeners as a first-class network contract, the "right way" for Docker/K8s: ```yaml # each caller gets an address it can actually reach KAFKA_ADVERTISED_LISTENERS: >- INTERNAL://kafka:9092, EXTERNAL://localhost:29092 # K8s: per-broker stable DNS or NodePort per broker — NEVER a shared Service VIP, # because the client must connect to the *specific* partition-leader broker. ``` And don't put `position()`/`endOffsets()` on a hot path or a naive readiness probe. Those calls make live RPCs to partition leaders; wiring them into a per-request health check turns one offline partition into an app-wide outage. Cache lag metrics from a dedicated poller with bounded timeouts, and gate readiness on `describeCluster(Duration)` — a cheap metadata probe — rather than on resolving positions for every partition. ## 7\. How to Prevent It Long-Term - **Alert on the broker signals, not just the client stack trace.** `OfflinePartitionsCount > 0` and `UnderReplicatedPartitions > 0` (JMX: `kafka.controller` / `kafka.server`) catch the leaderless-partition cause before consumers time out. A rising consumer-side `TimeoutException` rate with flat throughput is the fingerprint of a stuck position path. - **Ban `RF=1` in provisioning.** Enforce `replication-factor >= 3` and `min.insync.replicas=2` for consumed topics via topics-as-code (Strimzi `KafkaTopic` CRD, Terraform, or Spring `NewTopic` beans) reviewed in PRs. `auto.create.topics.enable=false` in every non-dev cluster. - **Test from the client's network namespace.** A CI smoke test that runs the consumer from a *separate* container (not `exec` inside the broker, where `kafka:9092` resolves for free) and asserts it consumes a record catches unroutable-leader misconfig that a broker-local test hides completely. - **Lint `advertised.listeners`.** Reject an internal hostname on an externally-reachable listener; that single check prevents the most common Docker/K8s form of this error. - **Bound every blocking consumer API.** Pass explicit `Duration` timeouts to `position`, `committed`, `endOffsets`, `beginningOffsets`, and `offsetsForTimes` so a bad leader fails in seconds and surfaces a clear message instead of a mysterious 60s stall. ## 8\. Key Takeaways - **"Position could be determined" = the offset-resolution step**, not fetch, commit, or group coordination. The consumer had partitions but couldn't learn where to start. - **It's routed to the partition leader.** `ListOffsets` and leader-epoch validation go to each partition's *leader* — bootstrap connectivity proves nothing about leader reachability. - **Two dominant causes:** a leaderless/offline partition (classic `RF=1` broker loss) or a leader advertised on an address the client can't route to (Docker/K8s listener trap). `kafka-topics.sh --describe` tells you which in one line. - **`default.api.timeout.ms` (60s) is the clock, not the cure.** Raising it only helps genuinely slow-but-healthy metadata; for offline or unroutable leaders it just delays the same failure. - **Design it out:** `RF>=3` \+ `min.insync.replicas=2`, advertised listeners as a real network contract, and bounded-timeout calls so a bad leader fails fast instead of hanging your consumer for a minute.