> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka OffsetOutOfRangeException: auto.offset.reset Explained - URL: https://www.ggorantala.dev/kafka-offsetoutofrangeexception-auto-offset-reset/ - Published: 2026-06-08T12:50:00.000Z - Updated: 2026-08-29T08:26:36.000Z - Description: Fix OffsetOutOfRangeException and NoOffsetForPartitionException in Kafka: why committed offsets vanish, what auto.offset.reset really does, and safe resets. - Author: Gopi Gorantala - Tags: kafka-errors, offsetoutofrangeexception, auto-offset-reset, kafka-consumer, spring-kafka, kafka-offsets, Java Your consumer was down over the weekend. Monday morning it won't start: ```text Exception in thread "main" org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {orders-0=1042} at org.apache.kafka.clients.consumer.internals.OffsetFetcherUtils.maybeThrowOffsetOutOfRangeException(OffsetFetcherUtils.java:170) at org.apache.kafka.clients.consumer.internals.FetchCollector.handleInitializeErrors(FetchCollector.java:311) at org.apache.kafka.clients.consumer.internals.FetchCollector.collectFetch(FetchCollector.java:114) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java...) ``` Its sibling, thrown when a brand-new consumer group has no committed offset at all: ```text org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partitions: [orders-0] ``` Both extend `InvalidOffsetException` and both come out of `poll()`. And here's the part that matters more than either stack trace: **if `auto.offset.reset` is `earliest` or `latest`, you never see an exception at all.** The consumer silently jumps — to the beginning (mass reprocessing) or to the end (data loss). The exception is the *good* outcome. The silent jump is the one that pages you three days later when finance asks where 40,000 orders went. Typical setup: kafka-clients 3.x/4.x (any version — this behavior is ancient and stable), Spring Kafka 3.x, brokers 3.9/4.0 in KRaft mode. Most common in production after a consumer outage that outlasted topic retention, and in POCs with aggressively short `retention.ms`. ## How to Reproduce It (step-by-step) Single-broker KRaft cluster: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.0 container_name: kafka ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093 KAFKA_LISTENERS: INTERNAL://:29092,CONTROLLER://:29093,EXTERNAL://:9092 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 # check retention every 10s so the repro is fast KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 10000 ``` Create a topic with brutal retention — 1-minute retention, 10-second segments: ```bash docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \ --create --topic orders --partitions 1 \ --config retention.ms=60000 --config segment.ms=10000 ``` Produce 1,000 records, consume them all, commit: ```bash docker exec kafka bash -c 'seq 1 1000 | /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 --topic orders' docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9092 --topic orders \ --group orders-svc --from-beginning --max-messages 1000 ``` Now simulate the weekend outage. The consumer is stopped; the committed offset for `orders-svc` is 1000\. Wait \~90 seconds, then produce a few more records — **this matters**: retention only deletes *rolled* segments, and the active segment rolls on the next append after `segment.ms` expires. No new appends, no roll, no deletion: ```bash sleep 90 docker exec kafka bash -c 'seq 1001 1010 | /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 --topic orders' sleep 30 # let the retention thread run ``` Verify the damage — log start offset has jumped past the committed offset: ```bash docker exec kafka /opt/kafka/bin/kafka-get-offsets.sh \ --bootstrap-server localhost:9092 --topic orders --time -2 # orders:0:1000 (or higher) — earliest available offset docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 --describe --group orders-svc # CURRENT-OFFSET 1000, but LOG-START past it once more segments age out ``` Restart the consumer with the reset policy that refuses to guess: ```java // kafka-clients 3.9.0 Properties props = new Properties(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-svc"); props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none"); // <-- the honest setting props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); try (KafkaConsumer consumer = new KafkaConsumer<>(props)) { consumer.subscribe(List.of("orders")); while (true) { consumer.poll(Duration.ofSeconds(1)); // throws OffsetOutOfRangeException } } ``` If the committed offset survived but points below the log start offset, you get `OffsetOutOfRangeException`. Run the same code with a fresh `group.id` that has never committed, and you get `NoOffsetForPartitionException` instead. ## Why It Happens — Surface Level The consumer resumed from its committed offset (1042 in the trace above), sent a Fetch for that position, and the broker answered `OFFSET_OUT_OF_RANGE` because the log now starts at a higher offset — retention deleted the segments containing 1042\. The consumer checks `auto.offset.reset` to decide what to do. `none` means "throw and let a human decide," so it throws. `NoOffsetForPartitionException` is the same decision point reached from the other side: there's no committed offset for this group at all (new group, or the committed offset *expired* — more on that below), and `none` forbids guessing a starting position. ## Why It Happens — Under the Hood Three independent clocks can strand your committed offset, and conflating them causes wrong fixes. **1\. Log retention advances the log start offset.** The retention thread (running every `log.retention.check.interval.ms`, default 5 min) deletes rolled segments whose newest timestamp exceeds `retention.ms` (default 7 days) or when the partition exceeds `retention.bytes`. Deleting a segment advances the partition's *log start offset*. Committed offsets live in `__consumer_offsets` — a completely separate topic — so nothing updates them when data is deleted. Your offset becomes a dangling pointer. `DeleteRecordsRequest` (`kafka-delete-records.sh`) and compaction-with-delete cleanup do the same thing. **2\. Committed offsets themselves expire.** `offsets.retention.minutes` defaults to 10080 (7 days). Since KIP-211 (Kafka 2.1), the clock only starts when the group becomes *empty* — a live group's offsets never expire, which killed the old 0.x/1.x horror where a slow-but-alive consumer lost its offsets. But if your service is down (or your group uses manual `assign()` with no group membership) for longer than 7 days, the group coordinator deletes the offsets, and your restart behaves like a brand-new group: `NoOffsetForPartitionException` under `none`, silent jump otherwise. Note the trap: broker log retention of 30 days with default offset retention of 7 days means a two-week outage loses your *position* even though the *data* is still there. **3\. The fetch offset can also be *above* the log end offset.** Rarer, but real: after an unclean leader election, the new leader may have truncated data the old leader had. Your committed offset can point past the new leader's log end offset. Same `OFFSET_OUT_OF_RANGE` error code, very different root cause — check for `unclean.leader.election.enable=true` and leader-change log lines before blaming retention. Mechanically, the reset path is: `FetchCollector` sees `OFFSET_OUT_OF_RANGE` in the fetch response → checks the partition's reset strategy in `SubscriptionState` → for `earliest`/`latest` it fires a `ListOffsets` request with timestamp `-2` (earliest) or `-1` (latest) and repositions; for `none` it throws. The same strategy lookup handles the no-committed-offset case during group initialization. This is also why `earliest`/`latest` is invisible in application logs unless you log `ConsumerRebalanceListener` positions or watch the `Resetting offset for partition orders-0 to position ...` INFO line from `SubscriptionState`. One more misconception worth killing: `auto.offset.reset` does **not** mean "where the consumer starts." It means "where the consumer starts *only when there is no valid committed offset*." A group with valid commits ignores it entirely — which is why flipping it to `earliest` on a running group and bouncing the service does nothing, a mistake I've watched three different teams make during incidents. ## The Fix ### Immediate: reset offsets explicitly with the CLI Stop all consumers in the group first (the tool refuses to act on a non-empty group), then: ```bash # dry run — always look before you leap docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 --group orders-svc \ --topic orders --reset-offsets --to-earliest --dry-run # execute docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 --group orders-svc \ --topic orders --reset-offsets --to-earliest --execute ``` Pick the target deliberately: `--to-earliest` reprocesses everything still on disk (requires idempotent downstream processing), `--to-latest` accepts the gap and moves on, `--to-datetime 2026-07-06T00:00:00.000` restarts from a known-good point, `--shift-by -1000` rewinds relative. For loss-sensitive pipelines, `--to-datetime` at the outage start is usually the right call. ### Config: choose a reset policy on purpose The default is `latest`, which is the worst possible default for anything that matters — it converts "consumer was down too long" into silent data loss. ```diff # consumer.properties group.id=orders-svc -# auto.offset.reset unset -> defaults to latest (silent loss) +auto.offset.reset=none ``` Use `none` \+ alerting + the CLI runbook above for pipelines where skipping or double-processing data is an incident. Use `earliest` for idempotent consumers where reprocessing is safe and cheap. Use `latest` only for genuinely ephemeral reads — live dashboards, metrics samplers — where old data is worthless. ### Spring Kafka: fail loudly, not in a loop ```yaml spring: kafka: consumer: group-id: orders-svc auto-offset-reset: none ``` Beware: with `none`, the `InvalidOffsetException` escapes `poll()` inside the listener container, and the container's default behavior is to log and re-poll — an infinite error loop that looks alive to your orchestrator. Wire a `CommonContainerStoppingErrorHandler` fallback so the container stops and your health checks actually fail: ```java @Bean public DefaultErrorHandler errorHandler(KafkaTemplate template) { DefaultErrorHandler handler = new DefaultErrorHandler( new DeadLetterPublishingRecoverer(template), new FixedBackOff(1000L, 3)); // offset gone = ops decision, not a retry loop handler.addNotRetryableExceptions(InvalidOffsetException.class); return handler; } ``` For a hard stop instead of DLT routing, use `new CommonContainerStoppingErrorHandler()` directly — with liveness probes tied to container state, Kubernetes surfaces the failure instead of masking it. ### If the cause was offset expiry, not data expiry Align the retention clocks: ```diff # broker (or per-group via AdminClient in newer brokers) -offsets.retention.minutes=10080 +offsets.retention.minutes=43200 # 30 days, >= your longest topic retention ``` Rule of thumb: `offsets.retention.minutes` ≥ your longest consumer outage you intend to survive, and ideally ≥ topic `retention.ms`. There is no benefit to offsets outliving data, but plenty of pain the other way around. ## Best Practices & The Better Design The class of failure here is *implicit position management* — letting a config default decide where your consumer reads after an anomaly. The better design makes position an explicit, observable decision: ```java props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none"); consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection parts) {} @Override public void onPartitionsAssigned(Collection parts) { // positions are valid here or poll() would have thrown; // log them so every restart leaves an audit trail parts.forEach(tp -> log.info("Resuming {} at offset {}", tp, consumer.position(tp))); } }); try { while (running) { ConsumerRecords records = consumer.poll(Duration.ofMillis(500)); process(records); consumer.commitSync(); } } catch (InvalidOffsetException e) { // covers both OffsetOutOfRange and NoOffsetForPartition log.error("Offset position lost for {} — halting for operator reset. " + "Runbook: kafka-consumer-groups.sh --reset-offsets --to-datetime ", e.partitions(), e); alerting.page("kafka-offset-reset-required", e.partitions().toString()); throw e; // crash; do NOT auto-reset in loss-sensitive services } ``` Complement it with an outage-survival budget written down as config policy: topic `retention.ms` sized to cover your worst-case consumer downtime plus reprocessing headroom (7 days of data at peak throughput is your recovery window, treat it like one), `offsets.retention.minutes` matched to it, and consumers idempotent end-to-end so `--to-earliest`/`--to-datetime` resets are boring instead of terrifying. Idempotency is what turns this whole error class from "data-integrity incident" into "run the runbook, go back to lunch." ## How to Prevent It Long-Term **Alert on consumer lag *and* group emptiness.** Lag monitoring (Burrow, `kafka_consumergroup_lag` via the exporter, or your vendor's equivalent) catches the growing gap, but the killer metric is *group has no active members for > N hours* — that's the precursor to both offset expiry and retention overrun. Page on it long before 7 days. **Alert on proximity to log start.** `records-lag-max` tells you distance from the log *end*; the dangerous distance is committed offset minus log *start* offset. A periodic job comparing `kafka-get-offsets.sh --time -2` output against `kafka-consumer-groups.sh --describe` per group gives you "hours until my offset falls off the log" — alert when it drops below a day. **CI/config conventions.** Lint consumer configs in CI: any production consumer without an explicit `auto.offset.reset` fails the build. The default-`latest` foot-gun should be impossible to ship unnoticed. **Chaos-test the outage path.** Once a quarter, stop a staging consumer group past a short-retention topic's window and verify: the alert fires, the runbook works, the reset target is correct, downstream dedup holds under `--to-earliest`. Every team I've seen do this found a broken runbook the first time. ## Key Takeaways - `auto.offset.reset` only applies when there's **no valid committed offset** — it does not control where a healthy group starts, and changing it on a running group does nothing. - The exception (`OffsetOutOfRangeException` / `NoOffsetForPartitionException`) only appears with `none`; `earliest` silently reprocesses, `latest` silently loses data. For anything loss-sensitive, `none` \+ runbook beats both. - Committed offsets and log data expire on **separate clocks**: `offsets.retention.minutes` (7 days, counted from group emptiness since KIP-211) vs `retention.ms` (7 days, per segment). Either one can strand you; align them. - Recover with `kafka-consumer-groups.sh --reset-offsets` (`--dry-run` first, group must be stopped) — `--to-datetime` at outage start is usually the right target. - Make consumers idempotent so offset resets are routine operations, not data-integrity incidents.