> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka TimeoutException: Expiring N record(s) — Cause and Fix - URL: https://www.ggorantala.dev/kafka-timeoutexception-expiring-records/ - Published: 2026-06-25T14:16:00.000Z - Updated: 2026-08-29T08:22:03.000Z - Description: Kafka producer failing with TimeoutException: Expiring N record(s): ms has passed since batch creation? What delivery.timeout.ms really does — and the fix. - Author: Gopi Gorantala - Tags: kafka-errors, TimeoutException, kafka-producer, delivery-timeout-ms, spring-kafka, Java, event-streaming Your producer's send callbacks (or `Future.get()`) start failing with: ```text org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for orders-3:120000 ms has passed since batch creation ``` Older clients (< 2.1) word it slightly differently — same failure class: ```text org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for orders-0: 30022 ms has passed since batch creation plus linger time ``` In Spring Kafka it usually surfaces wrapped: ```text org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for orders-3:120000 ms has passed since batch creation ``` Typical setup: kafka-clients 3.x/4.x (`delivery.timeout.ms` default 120000), Spring Boot 3.x with Spring Kafka 3.x, against Docker Compose locally or a managed cluster (MSK, Confluent Cloud, Aiven) in production. Everything below is verified against kafka-clients 3.9.0; the mechanics are unchanged in 4.x. Two things to internalize before touching config: 1. **This exception is asynchronous.** `send()` already returned successfully. The records died later, inside the producer's background sender machinery. If you're fire-and-forgetting, you've been losing these records silently. 2. **This is not the `max.block.ms` timeout.** `TimeoutException: Topic orders not present in metadata after 60000 ms` comes from `send()` itself blocking — a different code path with different causes (usually connectivity/listeners). "…has passed since batch creation" means the producer *was* accepting records; it just couldn't deliver them in time. ## 2\. How to Reproduce It Single-broker KRaft cluster: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.0 container_name: kafka ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ``` ```bash docker compose up -d docker exec kafka /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:19092 \ --create --topic orders --partitions 3 --replication-factor 1 ``` Producer with a short delivery timeout so you don't wait two minutes per repro: ```java // kafka-clients 3.9.0 Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 10_000); // default 120000 props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 5_000); // default 30000 try (var producer = new KafkaProducer(props)) { for (int i = 0; i < 100_000; i++) { producer.send(new ProducerRecord<>("orders", "k" + i, "v" + i), (md, ex) -> { if (ex != null) System.err.println(ex); }); } producer.flush(); } ``` Start it, then freeze the broker mid-run: ```bash docker pause kafka ``` `docker pause` (SIGSTOP) is the right tool here — the TCP connection stays established, so the producer doesn't get a fast connection-refused; it just sees a broker that stops answering. Ten seconds later: ```text org.apache.kafka.common.errors.TimeoutException: Expiring 37 record(s) for orders-1:10000 ms has passed since batch creation ``` `docker unpause kafka` and the errors stop. Environment triggers that produce the same signature without an outage: a partition leader election taking longer than `delivery.timeout.ms` (broker restart with many partitions), broker disk saturation pushing produce latency into seconds, producer throughput exceeding what the connection can drain, or quota throttling on managed Kafka. ## 3\. Why It Happens — Surface Level Every record you `send()` goes into an in-memory batch, per partition. A batch gets a creation timestamp the moment it's allocated. From that moment, a countdown runs: `**delivery.timeout.ms**` (default 120000). If the batch has not been successfully acknowledged by the leader within that window — because the leader is down, unreachable, slow, or the queue in front of the batch is too long — the producer gives up on the whole batch, fails every record in it with this `TimeoutException`, and moves on. So the message is literally accurate: N records sat in a batch for `delivery.timeout.ms` milliseconds without getting acked. The question is *why* they sat that long, and there are exactly two families of answers: the broker side couldn't accept them (unavailability/slowness), or the producer side queued more than it could drain (overload). The fix differs completely between the two, which is why blindly raising the timeout is usually the wrong move. ## 4\. Why It Happens — Under the Hood The producer is two halves. Your application thread calls `send()`, which serializes the record and appends it to a `RecordAccumulator` — a map of partition → deque of `ProducerBatch`. A single background **Sender** thread drains ready batches, groups them into produce requests per broker, and manages retries. `delivery.timeout.ms` came from KIP-91 (Kafka 2.1) and replaced the old unbounded-retry chaos with one end-to-end budget covering the batch's *entire* life: ```text send() returns │ time waiting in accumulator (backpressure, leader unknown) │ + linger.ms │ + in-flight time (request.timeout.ms per attempt) │ + retry.backoff.ms between attempts ▼ ack from leader ── must all fit inside delivery.timeout.ms ``` Details that matter when you're debugging: - **`retries` is effectively dead as a knob.** Default is `Integer.MAX_VALUE`; the real retry bound is `delivery.timeout.ms`. A batch can fail *mid-retry*: each attempt gets `request.timeout.ms`, but the moment the total budget is spent, the Sender's expiry scan (`RecordAccumulator#expiredBatches`, run every poll loop) kills it regardless. - **Batches with no leader expire too.** If the partition is leaderless (election in progress, broker down), batches aren't even drainable — they sit in the accumulator aging until the deadline. This is why a 2-minute broker outage manifests as a wave of expired batches at exactly +120s. - **The config is validated at startup.** `delivery.timeout.ms` must be ≥ `linger.ms + request.timeout.ms`, or `KafkaProducer` throws `ConfigException` at construction. If you tune `request.timeout.ms` up, mind this arithmetic. - **Head-of-line blocking within a partition.** Batches per partition are delivered in order. One slow/retrying batch at the head of `orders-3`'s deque ages everything queued behind it — which is why the error often reports dozens of records across consecutive batches for the *same* partition. - **Idempotent producer interaction.** With `enable.idempotence=true` (default since 3.0), an expired batch means a sequence number the broker may never see. Clients ≥ 2.5 handle this with an epoch bump (KIP-360), so subsequent sends recover — but the expired records themselves are gone unless *you* re-send them. The producer will not resurrect an expired batch. One more distinction worth pinning down, because all three get pasted into the same search box: | Exception message | Thrown from | Meaning | | ----------------------------------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------- | | Expiring N record(s) … since batch creation | callback / future | Accepted but undeliverable within delivery.timeout.ms | | Topic X not present in metadata after N ms | send() | Metadata never arrived — connectivity/listener/topic problem (max.block.ms) | | Failed to allocate memory within the configured max blocking time | send() | buffer.memory full and didn't free within max.block.ms — you're overloaded and the accumulator is already backed up | If you see the third one alongside batch expiry, skip straight to the overload fixes below. ## 5\. The Fix Diagnose first — the error is a symptom with two distinct diseases. **Check broker-side health for the window in question:** leader elections (`kafka-leader-election.sh` events in the controller log), produce latency, disk. Check producer metrics if you have them: `record-error-rate`, `request-latency-avg`, `buffer-available-bytes`. ### Fix A: Broker was unavailable or slow (outage, leader election, disk) There is no producer config that makes an unreachable leader reachable. Fix the availability problem. What you *can* decide is how long the producer keeps trying before declaring failure — i.e., how much broker downtime you're willing to ride out in-memory: ```diff # producer.properties bootstrap.servers=broker1:9092,broker2:9092,broker3:9092 -delivery.timeout.ms=120000 +# survive a rolling restart / slow leader election without failing sends +delivery.timeout.ms=300000 +request.timeout.ms=30000 ``` Raising it is legitimate **only** when your application can tolerate the added worst-case latency and holds enough `buffer.memory` to absorb the backlog (records awaiting delivery live in that 32 MB default). A payment API that answers a client in 2s should *not* run a 5-minute delivery timeout — it should fail fast and surface the error. ### Fix B: Producer overload (throughput > drain rate) If brokers were healthy and you still expired batches, you queued faster than one Sender thread over N connections could deliver. Raising `delivery.timeout.ms` here just builds a longer queue that still overflows. Make each request carry more instead: ```diff # producer.properties -# defaults: batch.size=16384, linger.ms=0, compression.type=none +batch.size=65536 +linger.ms=10 +compression.type=lz4 ``` Fewer, bigger, compressed requests routinely buy 3–5× throughput on the same connection. If that's still not enough: check partition count (a single partition serializes everything through one deque), and scale to multiple producer instances — one `KafkaProducer` has exactly one Sender thread, and that thread does saturate. ### Fix C (mandatory regardless): stop fire-and-forgetting The config fixes reduce the error rate; this one stops it from being silent data loss: ```diff -producer.send(record); +producer.send(record, (metadata, exception) -> { + if (exception instanceof TimeoutException) { + // batch expired — record was NOT written. Re-send or persist to an outbox. + failedSends.add(record); + } else if (exception != null) { + log.error("Non-retriable send failure for key={}", record.key(), exception); + } +}); ``` Spring Kafka equivalent: ```java kafkaTemplate.send("orders", key, value) .whenComplete((result, ex) -> { if (ex != null) { // KafkaProducerException wrapping TimeoutException retryOrOutbox(key, value, ex); } }); ``` ## 6\. Best Practices & The Better Design The class of problem here is "producer holds unacknowledged business data in volatile memory with a deadline." The better design acknowledges that and puts durability on the correct side of the deadline. **Explicit timeout budget, written down, per producer.** Decide the arithmetic instead of inheriting defaults: ```properties # The producer's SLA, stated as config (kafka-clients 3.x / 4.x) acks=all enable.idempotence=true max.in.flight.requests.per.connection=5 # safe with idempotence, preserves ordering request.timeout.ms=30000 # one attempt retry.backoff.ms=100 delivery.timeout.ms=120000 # total budget: ~3-4 attempts + queue time linger.ms=10 batch.size=65536 compression.type=lz4 buffer.memory=67108864 # sized for delivery.timeout.ms worth of backlog max.block.ms=10000 # fail send() fast when buffer is full; don't hang app threads ``` The `buffer.memory` line deserves the comment: your buffer must hold `(peak throughput × delivery.timeout.ms)` bytes, or overload converts into blocked `send()` calls before batches even get the chance to expire. **Transactional outbox for anything money-shaped.** If the record's loss is unacceptable, the callback-and-retry pattern above is still best-effort — your process can die with `failedSends` in memory. Write the event to an outbox table in the same DB transaction as the business write, and have a relay (Debezium, or a polling publisher) own the Kafka delivery with its own retry loop. Batch expiry then becomes a lag alert, not data loss. **Fail fast at the edges, be patient in the middle.** Request-path producers (HTTP handlers): short `delivery.timeout.ms` (\~30s), short `max.block.ms`, propagate the failure to the caller. Async pipeline producers (CDC relays, ETL): long `delivery.timeout.ms` (5+ min), large buffer, ride out rolling restarts. Most production incidents I've debugged around this error trace back to one config profile being reused for both jobs. ## 7\. How to Prevent It Long-Term **Monitor the leading indicators, not just the exception.** The producer exposes everything you need via JMX/Micrometer under `producer-metrics`: alert on `record-error-rate > 0` (page-worthy — every unit is a lost record if uncaught), watch `record-retry-rate`, `buffer-available-bytes` (sustained decline = drain deficit), `request-latency-avg` (broker slowness before it becomes expiry), and `batch-size-avg` vs `batch.size` (whether batching is actually working). On the broker side, alert on `UnderReplicatedPartitions` and produce-request p99 — those degrade minutes before producers start expiring batches. **Validate the timeout arithmetic in CI.** A unit test that constructs your `ProducerConfig` catches the `delivery.timeout.ms < linger.ms + request.timeout.ms` `ConfigException` before deploy, and is a natural place to assert your team's conventions (idempotence on, `acks=all`, sane `max.block.ms`). **Chaos-test the exact failure.** The `docker pause` repro from section 2 belongs in your integration suite (Testcontainers can pause containers programmatically). Assert that a paused-then-unpaused broker results in zero unhandled send failures — that single test verifies your callback handling, buffer sizing, and timeout budget together. **Load-test to the expiry cliff.** Push a canary producer until batches expire and record the throughput ceiling per instance. Capacity planning against a measured number beats discovering the ceiling during your Black Friday. ## 8\. Key Takeaways - `Expiring N record(s) … since batch creation` means records were accepted by `send()` but not acknowledged within `delivery.timeout.ms` (default 120s) — it's asynchronous, and without a callback it's silent data loss. - The end-to-end budget is `delivery.timeout.ms`; `retries` is effectively meaningless since KIP-91\. Keep `delivery.timeout.ms ≥ linger.ms + request.timeout.ms` or the producer won't even start. - Two root-cause families: broker unavailable/slow (fix availability; optionally extend the budget) vs producer overload (fix batching/compression/partitions — a longer timeout just delays the same failure). - Never call `send()` without a callback in production. For unlosable data, use a transactional outbox so Kafka delivery failure degrades to lag instead of loss. - Alert on `record-error-rate`, `buffer-available-bytes`, and broker produce latency — they fire before the two-minute fuse burns down.