Kafka TimeoutException: Expiring N record(s) — Cause and Fix

Your producer's send callbacks (or Future.get()) start failing with:

org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for orders-3:120000 ms has passed since batch creation

Older clients (< 2.1) word it slightly differently — same failure class:

org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for orders-0: 30022 ms has passed since batch creation plus linger time

In Spring Kafka it usually surfaces wrapped:

org.springframework.kafka.core.KafkaProducerException: Failed to send;
 nested exception is org.apache.kafka.common.errors.TimeoutException:
 Expiring 15 record(s) for orders-3:120000 ms has passed since batch creation

Typical setup: kafka-clients 3.x/4.x (delivery.timeout.ms default 120000), Spring Boot 3.x with Spring Kafka 3.x, against Docker Compose locally or a managed cluster (MSK, Confluent Cloud, Aiven) in production. Everything below is verified against kafka-clients 3.9.0; the mechanics are unchanged in 4.x.

Two things to internalize before touching config:

1. This exception is asynchronous. send() already returned successfully. The records died later, inside the producer's background sender machinery. If you're fire-and-forgetting, you've been losing these records silently.
2. This is not the max.block.ms timeout. TimeoutException: Topic orders not present in metadata after 60000 ms comes from send() itself blocking — a different code path with different causes (usually connectivity/listeners). "…has passed since batch creation" means the producer was accepting records; it just couldn't deliver them in time.

2. How to Reproduce It

Single-broker KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:19092 \
  --create --topic orders --partitions 3 --replication-factor 1

Producer with a short delivery timeout so you don't wait two minutes per repro:

// kafka-clients 3.9.0
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 10_000); // default 120000
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 5_000);   // default 30000

try (var producer = new KafkaProducer<String, String>(props)) {
    for (int i = 0; i < 100_000; i++) {
        producer.send(new ProducerRecord<>("orders", "k" + i, "v" + i),
            (md, ex) -> { if (ex != null) System.err.println(ex); });
    }
    producer.flush();
}

Start it, then freeze the broker mid-run:

docker pause kafka

docker pause (SIGSTOP) is the right tool here — the TCP connection stays established, so the producer doesn't get a fast connection-refused; it just sees a broker that stops answering. Ten seconds later:

org.apache.kafka.common.errors.TimeoutException: Expiring 37 record(s) for orders-1:10000 ms has passed since batch creation

docker unpause kafka and the errors stop. Environment triggers that produce the same signature without an outage: a partition leader election taking longer than delivery.timeout.ms (broker restart with many partitions), broker disk saturation pushing produce latency into seconds, producer throughput exceeding what the connection can drain, or quota throttling on managed Kafka.

3. Why It Happens — Surface Level

Every record you send() goes into an in-memory batch, per partition. A batch gets a creation timestamp the moment it's allocated. From that moment, a countdown runs: delivery.timeout.ms (default 120000). If the batch has not been successfully acknowledged by the leader within that window — because the leader is down, unreachable, slow, or the queue in front of the batch is too long — the producer gives up on the whole batch, fails every record in it with this TimeoutException, and moves on.

So the message is literally accurate: N records sat in a batch for delivery.timeout.ms milliseconds without getting acked. The question is why they sat that long, and there are exactly two families of answers: the broker side couldn't accept them (unavailability/slowness), or the producer side queued more than it could drain (overload). The fix differs completely between the two, which is why blindly raising the timeout is usually the wrong move.

4. Why It Happens — Under the Hood

The producer is two halves. Your application thread calls send(), which serializes the record and appends it to a RecordAccumulator — a map of partition → deque of ProducerBatch. A single background Sender thread drains ready batches, groups them into produce requests per broker, and manages retries.

delivery.timeout.ms came from KIP-91 (Kafka 2.1) and replaced the old unbounded-retry chaos with one end-to-end budget covering the batch's entire life:

send() returns
   │  time waiting in accumulator (backpressure, leader unknown)
   │  + linger.ms
   │  + in-flight time (request.timeout.ms per attempt)
   │  + retry.backoff.ms between attempts
   ▼
ack from leader  ──  must all fit inside delivery.timeout.ms

Details that matter when you're debugging:

• retries is effectively dead as a knob. Default is Integer.MAX_VALUE; the real retry bound is delivery.timeout.ms. A batch can fail mid-retry: each attempt gets request.timeout.ms, but the moment the total budget is spent, the Sender's expiry scan (RecordAccumulator#expiredBatches, run every poll loop) kills it regardless.
• Batches with no leader expire too. If the partition is leaderless (election in progress, broker down), batches aren't even drainable — they sit in the accumulator aging until the deadline. This is why a 2-minute broker outage manifests as a wave of expired batches at exactly +120s.
• The config is validated at startup. delivery.timeout.ms must be ≥ linger.ms + request.timeout.ms, or KafkaProducer throws ConfigException at construction. If you tune request.timeout.ms up, mind this arithmetic.
• Head-of-line blocking within a partition. Batches per partition are delivered in order. One slow/retrying batch at the head of orders-3's deque ages everything queued behind it — which is why the error often reports dozens of records across consecutive batches for the same partition.
• Idempotent producer interaction. With enable.idempotence=true (default since 3.0), an expired batch means a sequence number the broker may never see. Clients ≥ 2.5 handle this with an epoch bump (KIP-360), so subsequent sends recover — but the expired records themselves are gone unless you re-send them. The producer will not resurrect an expired batch.

One more distinction worth pinning down, because all three get pasted into the same search box:

If you see the third one alongside batch expiry, skip straight to the overload fixes below.

5. The Fix

Diagnose first — the error is a symptom with two distinct diseases.

Check broker-side health for the window in question: leader elections (kafka-leader-election.sh events in the controller log), produce latency, disk. Check producer metrics if you have them: record-error-rate, request-latency-avg, buffer-available-bytes.

Fix A: Broker was unavailable or slow (outage, leader election, disk)

There is no producer config that makes an unreachable leader reachable. Fix the availability problem. What you can decide is how long the producer keeps trying before declaring failure — i.e., how much broker downtime you're willing to ride out in-memory:

 # producer.properties
 bootstrap.servers=broker1:9092,broker2:9092,broker3:9092
-delivery.timeout.ms=120000
+# survive a rolling restart / slow leader election without failing sends
+delivery.timeout.ms=300000
+request.timeout.ms=30000

Raising it is legitimate only when your application can tolerate the added worst-case latency and holds enough buffer.memory to absorb the backlog (records awaiting delivery live in that 32 MB default). A payment API that answers a client in 2s should not run a 5-minute delivery timeout — it should fail fast and surface the error.

Fix B: Producer overload (throughput > drain rate)

If brokers were healthy and you still expired batches, you queued faster than one Sender thread over N connections could deliver. Raising delivery.timeout.ms here just builds a longer queue that still overflows. Make each request carry more instead:

 # producer.properties
-# defaults: batch.size=16384, linger.ms=0, compression.type=none
+batch.size=65536
+linger.ms=10
+compression.type=lz4

Fewer, bigger, compressed requests routinely buy 3–5× throughput on the same connection. If that's still not enough: check partition count (a single partition serializes everything through one deque), and scale to multiple producer instances — one KafkaProducer has exactly one Sender thread, and that thread does saturate.

Fix C (mandatory regardless): stop fire-and-forgetting

The config fixes reduce the error rate; this one stops it from being silent data loss:

-producer.send(record);
+producer.send(record, (metadata, exception) -> {
+    if (exception instanceof TimeoutException) {
+        // batch expired — record was NOT written. Re-send or persist to an outbox.
+        failedSends.add(record);
+    } else if (exception != null) {
+        log.error("Non-retriable send failure for key={}", record.key(), exception);
+    }
+});

Spring Kafka equivalent:

kafkaTemplate.send("orders", key, value)
    .whenComplete((result, ex) -> {
        if (ex != null) {
            // KafkaProducerException wrapping TimeoutException
            retryOrOutbox(key, value, ex);
        }
    });

6. Best Practices & The Better Design

The class of problem here is "producer holds unacknowledged business data in volatile memory with a deadline." The better design acknowledges that and puts durability on the correct side of the deadline.

Explicit timeout budget, written down, per producer. Decide the arithmetic instead of inheriting defaults:

# The producer's SLA, stated as config (kafka-clients 3.x / 4.x)
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5   # safe with idempotence, preserves ordering
request.timeout.ms=30000                  # one attempt
retry.backoff.ms=100
delivery.timeout.ms=120000                # total budget: ~3-4 attempts + queue time
linger.ms=10
batch.size=65536
compression.type=lz4
buffer.memory=67108864                    # sized for delivery.timeout.ms worth of backlog
max.block.ms=10000                        # fail send() fast when buffer is full; don't hang app threads

The buffer.memory line deserves the comment: your buffer must hold (peak throughput × delivery.timeout.ms) bytes, or overload converts into blocked send() calls before batches even get the chance to expire.

Transactional outbox for anything money-shaped. If the record's loss is unacceptable, the callback-and-retry pattern above is still best-effort — your process can die with failedSends in memory. Write the event to an outbox table in the same DB transaction as the business write, and have a relay (Debezium, or a polling publisher) own the Kafka delivery with its own retry loop. Batch expiry then becomes a lag alert, not data loss.

Fail fast at the edges, be patient in the middle. Request-path producers (HTTP handlers): short delivery.timeout.ms (~30s), short max.block.ms, propagate the failure to the caller. Async pipeline producers (CDC relays, ETL): long delivery.timeout.ms (5+ min), large buffer, ride out rolling restarts. Most production incidents I've debugged around this error trace back to one config profile being reused for both jobs.

7. How to Prevent It Long-Term

Monitor the leading indicators, not just the exception. The producer exposes everything you need via JMX/Micrometer under producer-metrics: alert on record-error-rate > 0 (page-worthy — every unit is a lost record if uncaught), watch record-retry-rate, buffer-available-bytes (sustained decline = drain deficit), request-latency-avg (broker slowness before it becomes expiry), and batch-size-avg vs batch.size (whether batching is actually working). On the broker side, alert on UnderReplicatedPartitions and produce-request p99 — those degrade minutes before producers start expiring batches.

Validate the timeout arithmetic in CI. A unit test that constructs your ProducerConfig catches the delivery.timeout.ms < linger.ms + request.timeout.ms ConfigException before deploy, and is a natural place to assert your team's conventions (idempotence on, acks=all, sane max.block.ms).

Chaos-test the exact failure. The docker pause repro from section 2 belongs in your integration suite (Testcontainers can pause containers programmatically). Assert that a paused-then-unpaused broker results in zero unhandled send failures — that single test verifies your callback handling, buffer sizing, and timeout budget together.

Load-test to the expiry cliff. Push a canary producer until batches expire and record the throughput ceiling per instance. Capacity planning against a measured number beats discovering the ceiling during your Black Friday.

8. Key Takeaways

• Expiring N record(s) … since batch creation means records were accepted by send() but not acknowledged within delivery.timeout.ms (default 120s) — it's asynchronous, and without a callback it's silent data loss.
• The end-to-end budget is delivery.timeout.ms; retries is effectively meaningless since KIP-91. Keep delivery.timeout.ms ≥ linger.ms + request.timeout.ms or the producer won't even start.
• Two root-cause families: broker unavailable/slow (fix availability; optionally extend the budget) vs producer overload (fix batching/compression/partitions — a longer timeout just delays the same failure).
• Never call send() without a callback in production. For unlosable data, use a transactional outbox so Kafka delivery failure degrades to lag instead of loss.
• Alert on record-error-rate, buffer-available-bytes, and broker produce latency — they fire before the two-minute fuse burns down.