Kafka Timeout Before Position for Partition Could Be Determined

Your consumer starts, joins the group, gets partitions assigned — and then, 60 seconds later, dies with this:

org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired
before the position for partition orders-3 could be determined

No data consumed. No offset committed. The consumer had partitions but never figured out where to start reading them. This is not a networking flake you retry past — it's a metadata/routing failure on the position-resolution path, and it will loop forever until you fix the underlying leader or listener problem.

This article is specifically about the position timeout, which is mechanically distinct from the producer-side "not present in metadata" timeout, the group-coordinator "Marking coordinator dead" loop, and OffsetOutOfRangeException. Same word "timeout," completely different code path.

1. The Error

The exact message, as it lands in logs and gets pasted into search engines:

org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired
before the position for partition orders-3 could be determined

Framework-wrapped variants you'll also see:

# Spring Kafka container thread
org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is
  org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired
  before the position for partition orders-3 could be determined

# From endOffsets()/beginningOffsets() in a lag exporter or admin tool
org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired
  before the last committed offset for partitions [orders-3] could be determined

Environment where it bites:

• Client: kafka-clients 2.0 through 4.0 (classic KafkaConsumer and the KIP-848 AsyncKafkaConsumer share this contract), Spring Kafka 2.x–3.x, Kafka Connect, Kafka Streams, Beam/Druid Kafka connectors.
• Broker: any 2.x–4.0 line, KRaft or ZooKeeper.
• Setup: Docker/K8s local clusters with a misrouted per-partition leader, or any cluster running RF=1 topics where a broker went down.

The 60000ms in the message is not a coincidence — it's the default of default.api.timeout.ms (introduced by KIP-266 in Kafka 2.0). That number is your first clue about which knob is involved.

2. How to Reproduce It

The cleanest reproduction is an RF=1 topic whose leader broker is unreachable from the consumer. Single-broker KRaft compose:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "29092:29092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

Create the topic and a consumer group with a committed offset:

docker compose up -d

# create a 4-partition topic
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 4 --replication-factor 1

# produce a few records and consume once so the group has committed offsets
docker compose exec kafka bash -c \
  'seq 10 | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders'

docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders \
  --group demo --from-beginning --timeout-ms 5000

Now minimal Java consumer that resolves position explicitly (frameworks do this for you under the hood):

// kafka-clients 3.9.0
Properties p = new Properties();
p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
p.put(ConsumerConfig.GROUP_ID_CONFIG, "demo");
p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

try (KafkaConsumer<String, String> c = new KafkaConsumer<>(p)) {
    c.subscribe(List.of("orders"));
    c.poll(Duration.ofSeconds(2));                 // triggers join + assignment
    for (TopicPartition tp : c.assignment()) {
        System.out.println(tp + " @ " + c.position(tp));  // <-- blocks here
    }
}

Trigger the failure. Kill the only broker that leads orders, then run the consumer:

docker compose stop kafka

With the leader gone, position() (and the internal position resolution inside poll()) can never complete its ListOffsets round-trip. After default.api.timeout.ms = 60s, it throws.

The more common production trigger doesn't require killing anything: run the consumer from your host machine while the broker only advertises the internal address kafka:9092. Bootstrap to localhost:29092 succeeds, metadata comes back, but the per-partition leader is advertised as kafka:9092, which your host can't resolve. Position resolution starves against an unroutable leader — identical error, and nothing is actually "down."

3. Why It Happens — Surface Level

Before a consumer can fetch a single byte, it must establish a position for every assigned partition: the exact offset to read from next. That position comes from one of two places — the last committed offset (fetched from the group coordinator), or a reset offset derived from auto.offset.reset (fetched from the partition leader via a ListOffsets request). Either way, the client has to talk to a specific broker and get an answer.

If that broker can't be reached — because the partition has no leader, the leader is on a downed broker, or the leader's advertised address is unroutable from the client — the request never completes. The consumer keeps retrying inside the blocking call until default.api.timeout.ms (60s) elapses, then surfaces TimeoutException. The word "position" in the message is the tell: this is the offset-resolution step, not fetching, not committing, not group coordination.

4. Why It Happens — Under the Hood

Position resolution runs inside updateFetchPositions(), called on every poll() and directly by position(), endOffsets(), beginningOffsets(), and offsetsForTimes(). It has three sub-steps, each of which can stall:

1. refreshCommittedOffsetsIfNeeded — an OffsetFetch RPC to the group coordinator (the broker leading the relevant __consumer_offsets partition). If a committed offset exists, it becomes the position. Coordinator unreachable → this stalls (and overlaps the "Marking the coordinator dead" symptom, but here it surfaces as the position timeout because that's the caller).
2. resetPositionsIfNeeded — for partitions with no valid committed offset, a ListOffsets RPC to the partition leader resolves the earliest/latest offset per auto.offset.reset. This is the step that requires the partition leader specifically, and it's the one that starves when a leader is offline or unroutable.
3. validatePositionsIfNeeded — KIP-320 leader-epoch validation via OffsetsForLeaderEpoch, also sent to the partition leader, to detect truncation after an unclean leadership change. Another leader-dependent round-trip.

The critical detail: ListOffsets and OffsetsForLeaderEpoch are routed to the leader of each partition, resolved from cluster metadata. Bootstrap connectivity proves nothing about leader connectivity — they are independent connection paths to potentially different advertised addresses. That's why "the broker is up and my bootstrap works" and "position times out" are perfectly consistent states.

Two clocks govern the wait. request.timeout.ms (default 30000ms) bounds a single RPC attempt; on failure the client refreshes metadata and retries. default.api.timeout.ms (default 60000ms) bounds the whole blocking operation across all those retries. Because retries are time-bounded, not count-bounded, the consumer will happily re-send ListOffsets to a dead leader for a full 60 seconds before giving up. When a partition is genuinely leaderless (Leader: none, empty ISR after an RF=1 broker loss), metadata refresh returns the same "no leader" answer each cycle and the operation is doomed from the first attempt — it just takes 60s to admit it.

5. The Fix

First, identify which partition and whether it has a leader. Don't guess:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

Topic: orders  Partition: 3  Leader: none  Replicas: 2  Isr:        # offline — no leader
Topic: orders  Partition: 3  Leader: 2     Replicas: 2  Isr: 2      # leader exists — routing problem

• Leader: none → the partition is offline. Restore the broker hosting replica 2, or reassign leadership. On RF=1 there is no other replica to elect, so the data-bearing broker must come back (or you accept data loss and recreate). This is the single-replica trap.
• Leader: 2 but consumer still times out → the leader exists but the client can't route to its advertised address. It's a listener problem, not a broker problem.

For the advertised-listener case (the most common in Docker/K8s), give every caller a reachable address. Before/after:

# broker config — advertised.listeners is a network contract, not decoration
- KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+ KAFKA_LISTENERS: PLAINTEXT://:9092,EXTERNAL://:29092
+ KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:29092
+ KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT

# host-side consumer bootstraps on the listener whose advertised addr it can reach
- p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
+ p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");

Verify the leader is actually routable from where the client runs:

kafka-broker-api-versions.sh --bootstrap-server localhost:29092 | head
# each broker line prints its advertised host:port — must be reachable from THIS box

Fail fast instead of hanging 60s. If a downstream call (a lag exporter, a health check) calls position() or endOffsets(), pass an explicit Duration so it errors in seconds, not a minute:

- long end = consumer.endOffsets(partitions).get(tp);
+ // bound the call; catch and report unreachable leaders promptly
+ Map<TopicPartition, Long> end =
+     consumer.endOffsets(partitions, Duration.ofSeconds(5));

Raising default.api.timeout.ms is not the fix. It only helps the narrow case of a genuinely slow-but-healthy metadata path (a large cluster mid-election). If the leader is offline or unroutable, a bigger timeout just makes you wait longer for the same failure:

# stopgap ONLY for confirmed slow-metadata clusters — not for offline/unroutable leaders
- # default 60000
+ p.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 120000);

6. Best Practices & The Better Design

The whole class of failure disappears when no partition can be leaderless from a single broker loss and every advertised address is reachable by its clients.

Replication that survives one failure:

# never run RF=1 for anything a consumer must read continuously
kafka-topics.sh --bootstrap-server broker:9092 --create \
  --topic orders --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2

With RF=3, losing one broker triggers a sub-second leader election to a surviving in-sync replica; position resolution retries and finds the new leader well within the 60s budget. The transient burst self-heals — exactly what the timeout window is for. RF=1 gives it nothing to fall back to.

Listeners as a first-class network contract, the "right way" for Docker/K8s:

# each caller gets an address it can actually reach
KAFKA_ADVERTISED_LISTENERS: >-
  INTERNAL://kafka:9092,
  EXTERNAL://localhost:29092
# K8s: per-broker stable DNS or NodePort per broker — NEVER a shared Service VIP,
# because the client must connect to the *specific* partition-leader broker.

And don't put position()/endOffsets() on a hot path or a naive readiness probe. Those calls make live RPCs to partition leaders; wiring them into a per-request health check turns one offline partition into an app-wide outage. Cache lag metrics from a dedicated poller with bounded timeouts, and gate readiness on describeCluster(Duration) — a cheap metadata probe — rather than on resolving positions for every partition.

7. How to Prevent It Long-Term

• Alert on the broker signals, not just the client stack trace. OfflinePartitionsCount > 0 and UnderReplicatedPartitions > 0 (JMX: kafka.controller / kafka.server) catch the leaderless-partition cause before consumers time out. A rising consumer-side TimeoutException rate with flat throughput is the fingerprint of a stuck position path.
• Ban RF=1 in provisioning. Enforce replication-factor >= 3 and min.insync.replicas=2 for consumed topics via topics-as-code (Strimzi KafkaTopic CRD, Terraform, or Spring NewTopic beans) reviewed in PRs. auto.create.topics.enable=false in every non-dev cluster.
• Test from the client's network namespace. A CI smoke test that runs the consumer from a separate container (not exec inside the broker, where kafka:9092 resolves for free) and asserts it consumes a record catches unroutable-leader misconfig that a broker-local test hides completely.
• Lint advertised.listeners. Reject an internal hostname on an externally-reachable listener; that single check prevents the most common Docker/K8s form of this error.
• Bound every blocking consumer API. Pass explicit Duration timeouts to position, committed, endOffsets, beginningOffsets, and offsetsForTimes so a bad leader fails in seconds and surfaces a clear message instead of a mysterious 60s stall.

8. Key Takeaways

• "Position could be determined" = the offset-resolution step, not fetch, commit, or group coordination. The consumer had partitions but couldn't learn where to start.
• It's routed to the partition leader. ListOffsets and leader-epoch validation go to each partition's leader — bootstrap connectivity proves nothing about leader reachability.
• Two dominant causes: a leaderless/offline partition (classic RF=1 broker loss) or a leader advertised on an address the client can't route to (Docker/K8s listener trap). kafka-topics.sh --describe tells you which in one line.
• default.api.timeout.ms (60s) is the clock, not the cure. Raising it only helps genuinely slow-but-healthy metadata; for offline or unroutable leaders it just delays the same failure.
• Design it out: RF>=3 + min.insync.replicas=2, advertised listeners as a real network contract, and bounded-timeout calls so a bad leader fails fast instead of hanging your consumer for a minute.