Kafka OffsetOutOfRangeException: auto.offset.reset Explained

Your consumer was down over the weekend. Monday morning it won't start:

Exception in thread "main" org.apache.kafka.clients.consumer.OffsetOutOfRangeException:
Offsets out of range with no configured reset policy for partitions: {orders-0=1042}
	at org.apache.kafka.clients.consumer.internals.OffsetFetcherUtils.maybeThrowOffsetOutOfRangeException(OffsetFetcherUtils.java:170)
	at org.apache.kafka.clients.consumer.internals.FetchCollector.handleInitializeErrors(FetchCollector.java:311)
	at org.apache.kafka.clients.consumer.internals.FetchCollector.collectFetch(FetchCollector.java:114)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java...)

Its sibling, thrown when a brand-new consumer group has no committed offset at all:

org.apache.kafka.clients.consumer.NoOffsetForPartitionException:
Undefined offset with no reset policy for partitions: [orders-0]

Both extend InvalidOffsetException and both come out of poll(). And here's the part that matters more than either stack trace: if auto.offset.reset is earliest or latest, you never see an exception at all. The consumer silently jumps — to the beginning (mass reprocessing) or to the end (data loss). The exception is the good outcome. The silent jump is the one that pages you three days later when finance asks where 40,000 orders went.

Typical setup: kafka-clients 3.x/4.x (any version — this behavior is ancient and stable), Spring Kafka 3.x, brokers 3.9/4.0 in KRaft mode. Most common in production after a consumer outage that outlasted topic retention, and in POCs with aggressively short retention.ms.

How to Reproduce It (step-by-step)

Single-broker KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: INTERNAL://:29092,CONTROLLER://:29093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      # check retention every 10s so the repro is fast
      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 10000

Create a topic with brutal retention — 1-minute retention, 10-second segments:

docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 1 \
  --config retention.ms=60000 --config segment.ms=10000

Produce 1,000 records, consume them all, commit:

docker exec kafka bash -c 'seq 1 1000 | /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders'

docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders \
  --group orders-svc --from-beginning --max-messages 1000

Now simulate the weekend outage. The consumer is stopped; the committed offset for orders-svc is 1000. Wait ~90 seconds, then produce a few more records — this matters: retention only deletes rolled segments, and the active segment rolls on the next append after segment.ms expires. No new appends, no roll, no deletion:

sleep 90
docker exec kafka bash -c 'seq 1001 1010 | /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders'
sleep 30   # let the retention thread run

Verify the damage — log start offset has jumped past the committed offset:

docker exec kafka /opt/kafka/bin/kafka-get-offsets.sh \
  --bootstrap-server localhost:9092 --topic orders --time -2
# orders:0:1000  (or higher) — earliest available offset

docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group orders-svc
# CURRENT-OFFSET 1000, but LOG-START past it once more segments age out

Restart the consumer with the reset policy that refuses to guess:

// kafka-clients 3.9.0
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-svc");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");   // <-- the honest setting
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));
    while (true) {
        consumer.poll(Duration.ofSeconds(1));   // throws OffsetOutOfRangeException
    }
}

If the committed offset survived but points below the log start offset, you get OffsetOutOfRangeException. Run the same code with a fresh group.id that has never committed, and you get NoOffsetForPartitionException instead.

Why It Happens — Surface Level

The consumer resumed from its committed offset (1042 in the trace above), sent a Fetch for that position, and the broker answered OFFSET_OUT_OF_RANGE because the log now starts at a higher offset — retention deleted the segments containing 1042. The consumer checks auto.offset.reset to decide what to do. none means "throw and let a human decide," so it throws.

NoOffsetForPartitionException is the same decision point reached from the other side: there's no committed offset for this group at all (new group, or the committed offset expired — more on that below), and none forbids guessing a starting position.

Why It Happens — Under the Hood

Three independent clocks can strand your committed offset, and conflating them causes wrong fixes.

1. Log retention advances the log start offset. The retention thread (running every log.retention.check.interval.ms, default 5 min) deletes rolled segments whose newest timestamp exceeds retention.ms (default 7 days) or when the partition exceeds retention.bytes. Deleting a segment advances the partition's log start offset. Committed offsets live in __consumer_offsets — a completely separate topic — so nothing updates them when data is deleted. Your offset becomes a dangling pointer. DeleteRecordsRequest (kafka-delete-records.sh) and compaction-with-delete cleanup do the same thing.

2. Committed offsets themselves expire. offsets.retention.minutes defaults to 10080 (7 days). Since KIP-211 (Kafka 2.1), the clock only starts when the group becomes empty — a live group's offsets never expire, which killed the old 0.x/1.x horror where a slow-but-alive consumer lost its offsets. But if your service is down (or your group uses manual assign() with no group membership) for longer than 7 days, the group coordinator deletes the offsets, and your restart behaves like a brand-new group: NoOffsetForPartitionException under none, silent jump otherwise. Note the trap: broker log retention of 30 days with default offset retention of 7 days means a two-week outage loses your position even though the data is still there.

3. The fetch offset can also be above the log end offset. Rarer, but real: after an unclean leader election, the new leader may have truncated data the old leader had. Your committed offset can point past the new leader's log end offset. Same OFFSET_OUT_OF_RANGE error code, very different root cause — check for unclean.leader.election.enable=true and leader-change log lines before blaming retention.

Mechanically, the reset path is: FetchCollector sees OFFSET_OUT_OF_RANGE in the fetch response → checks the partition's reset strategy in SubscriptionState → for earliest/latest it fires a ListOffsets request with timestamp -2 (earliest) or -1 (latest) and repositions; for none it throws. The same strategy lookup handles the no-committed-offset case during group initialization. This is also why earliest/latest is invisible in application logs unless you log ConsumerRebalanceListener positions or watch the Resetting offset for partition orders-0 to position ... INFO line from SubscriptionState.

One more misconception worth killing: auto.offset.reset does not mean "where the consumer starts." It means "where the consumer starts only when there is no valid committed offset." A group with valid commits ignores it entirely — which is why flipping it to earliest on a running group and bouncing the service does nothing, a mistake I've watched three different teams make during incidents.

The Fix

Immediate: reset offsets explicitly with the CLI

Stop all consumers in the group first (the tool refuses to act on a non-empty group), then:

# dry run — always look before you leap
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --group orders-svc \
  --topic orders --reset-offsets --to-earliest --dry-run

# execute
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --group orders-svc \
  --topic orders --reset-offsets --to-earliest --execute

Pick the target deliberately: --to-earliest reprocesses everything still on disk (requires idempotent downstream processing), --to-latest accepts the gap and moves on, --to-datetime 2026-07-06T00:00:00.000 restarts from a known-good point, --shift-by -1000 rewinds relative. For loss-sensitive pipelines, --to-datetime at the outage start is usually the right call.

Config: choose a reset policy on purpose

The default is latest, which is the worst possible default for anything that matters — it converts "consumer was down too long" into silent data loss.

 # consumer.properties
 group.id=orders-svc
-# auto.offset.reset unset -> defaults to latest (silent loss)
+auto.offset.reset=none

Use none + alerting + the CLI runbook above for pipelines where skipping or double-processing data is an incident. Use earliest for idempotent consumers where reprocessing is safe and cheap. Use latest only for genuinely ephemeral reads — live dashboards, metrics samplers — where old data is worthless.

Spring Kafka: fail loudly, not in a loop

spring:
  kafka:
    consumer:
      group-id: orders-svc
      auto-offset-reset: none

Beware: with none, the InvalidOffsetException escapes poll() inside the listener container, and the container's default behavior is to log and re-poll — an infinite error loop that looks alive to your orchestrator. Wire a CommonContainerStoppingErrorHandler fallback so the container stops and your health checks actually fail:

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    DefaultErrorHandler handler = new DefaultErrorHandler(
        new DeadLetterPublishingRecoverer(template), new FixedBackOff(1000L, 3));
    // offset gone = ops decision, not a retry loop
    handler.addNotRetryableExceptions(InvalidOffsetException.class);
    return handler;
}

For a hard stop instead of DLT routing, use new CommonContainerStoppingErrorHandler() directly — with liveness probes tied to container state, Kubernetes surfaces the failure instead of masking it.

If the cause was offset expiry, not data expiry

Align the retention clocks:

 # broker (or per-group via AdminClient in newer brokers)
-offsets.retention.minutes=10080
+offsets.retention.minutes=43200   # 30 days, >= your longest topic retention

Rule of thumb: offsets.retention.minutes ≥ your longest consumer outage you intend to survive, and ideally ≥ topic retention.ms. There is no benefit to offsets outliving data, but plenty of pain the other way around.

Best Practices & The Better Design

The class of failure here is implicit position management — letting a config default decide where your consumer reads after an anomaly. The better design makes position an explicit, observable decision:

props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");

consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
    @Override public void onPartitionsRevoked(Collection<TopicPartition> parts) {}
    @Override public void onPartitionsAssigned(Collection<TopicPartition> parts) {
        // positions are valid here or poll() would have thrown;
        // log them so every restart leaves an audit trail
        parts.forEach(tp -> log.info("Resuming {} at offset {}", tp, consumer.position(tp)));
    }
});

try {
    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        process(records);
        consumer.commitSync();
    }
} catch (InvalidOffsetException e) {
    // covers both OffsetOutOfRange and NoOffsetForPartition
    log.error("Offset position lost for {} — halting for operator reset. " +
              "Runbook: kafka-consumer-groups.sh --reset-offsets --to-datetime <outage-start>",
              e.partitions(), e);
    alerting.page("kafka-offset-reset-required", e.partitions().toString());
    throw e;   // crash; do NOT auto-reset in loss-sensitive services
}

Complement it with an outage-survival budget written down as config policy: topic retention.ms sized to cover your worst-case consumer downtime plus reprocessing headroom (7 days of data at peak throughput is your recovery window, treat it like one), offsets.retention.minutes matched to it, and consumers idempotent end-to-end so --to-earliest/--to-datetime resets are boring instead of terrifying. Idempotency is what turns this whole error class from "data-integrity incident" into "run the runbook, go back to lunch."

How to Prevent It Long-Term

Alert on consumer lag and group emptiness. Lag monitoring (Burrow, kafka_consumergroup_lag via the exporter, or your vendor's equivalent) catches the growing gap, but the killer metric is group has no active members for > N hours — that's the precursor to both offset expiry and retention overrun. Page on it long before 7 days.

Alert on proximity to log start. records-lag-max tells you distance from the log end; the dangerous distance is committed offset minus log start offset. A periodic job comparing kafka-get-offsets.sh --time -2 output against kafka-consumer-groups.sh --describe per group gives you "hours until my offset falls off the log" — alert when it drops below a day.

CI/config conventions. Lint consumer configs in CI: any production consumer without an explicit auto.offset.reset fails the build. The default-latest foot-gun should be impossible to ship unnoticed.

Chaos-test the outage path. Once a quarter, stop a staging consumer group past a short-retention topic's window and verify: the alert fires, the runbook works, the reset target is correct, downstream dedup holds under --to-earliest. Every team I've seen do this found a broken runbook the first time.

Key Takeaways

• auto.offset.reset only applies when there's no valid committed offset — it does not control where a healthy group starts, and changing it on a running group does nothing.
• The exception (OffsetOutOfRangeException / NoOffsetForPartitionException) only appears with none; earliest silently reprocesses, latest silently loses data. For anything loss-sensitive, none + runbook beats both.
• Committed offsets and log data expire on separate clocks: offsets.retention.minutes (7 days, counted from group emptiness since KIP-211) vs retention.ms (7 days, per segment). Either one can strand you; align them.
• Recover with kafka-consumer-groups.sh --reset-offsets (--dry-run first, group must be stopped) — --to-datetime at outage start is usually the right target.
• Make consumers idempotent so offset resets are routine operations, not data-integrity incidents.