Kafka BufferExhaustedException: Fix Failed to Allocate Memory

1. The Error

Your producer thread — often every producer thread at once — starts throwing this out of send():

org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 16384 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 16384 bytes
	at org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:132)
	at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:295)
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1054)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:947)

On kafka-clients older than 2.8 the same failure surfaces as a plain TimeoutException with the identical Failed to allocate memory within the configured max blocking time message. Since 2.8, BufferExhaustedException extends TimeoutException, so old catch (TimeoutException) blocks still trigger — which is exactly why many teams mistake this for the other producer timeout.

Do not confuse it with these two:

• TimeoutException: Topic X not present in metadata after 60000 ms — metadata wait, also gated by max.block.ms, usually connectivity/listeners.
• TimeoutException: Expiring N record(s) for topic-0: 120000 ms has passed since batch creation — batch already accepted, expired later under delivery.timeout.ms.

BufferExhaustedException is earlier in the pipeline than both: send() couldn't even hand the record to the accumulator because the producer's buffer.memory pool is full and stayed full for max.block.ms.

Typical setup where it bites: kafka-clients 3.x (any), Spring Boot 3.3.x with spring-kafka 3.2.x, high-throughput batch jobs or Kafka-connected APIs under burst load, and — the classic — a broker that slowed down or lost ISR while the app kept producing at full rate.

2. How to Reproduce It

Single KRaft broker:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.8.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:19092,CONTROLLER://0.0.0.0:9093,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:19092 --create --topic orders --partitions 3

Producer with a deliberately small pool and short block time (kafka-clients 3.8.0):

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class BufferExhaustionRepro {
    public static void main(String[] args) {
        Properties p = new Properties();
        p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
              "org.apache.kafka.common.serialization.StringSerializer");
        p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
              "org.apache.kafka.common.serialization.StringSerializer");
        p.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 1_048_576);  // 1 MB, default 32 MB
        p.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 2_000);       // default 60000

        String payload = "x".repeat(1024); // 1 KB records
        try (Producer<String, String> producer = new KafkaProducer<>(p)) {
            for (int i = 0; i < 1_000_000; i++) {
                producer.send(new ProducerRecord<>("orders", "k" + i, payload));
            }
        }
    }
}

Start it, then freeze the broker so the sender thread can't drain the accumulator:

docker pause kafka

Within a few seconds the 1 MB pool is full, send() blocks, and after 2 s you get BufferExhaustedException. docker unpause kafka and it recovers. That's the whole failure mode in miniature: inflow rate exceeded drain rate for longer than the buffer could absorb.

In production nobody sets buffer.memory to 1 MB — the same math just plays out at 32 MB with a traffic spike, a slow ISR follower, a leader failover, or a min.insync.replicas stall doing the job of docker pause.

3. Why It Happens — Surface Level

KafkaProducer.send() is asynchronous, but it isn't unbounded. Every serialized record must get space in a shared, fixed-size memory pool (buffer.memory, default 32 MB) before send() returns. A background sender thread drains that pool to the brokers. If your application appends faster than the sender can ship — because of a burst, a slow or unavailable broker, or too little batching — the pool fills. send() then blocks up to max.block.ms (default 60 s) waiting for a batch to complete and release memory. If nothing frees up in time, you get BufferExhaustedException.

It's backpressure, surfaced as an exception. The producer is telling you the downstream can't keep up with the upstream, and its shock absorber is full.

4. Why It Happens — Under the Hood

Inside the producer, RecordAccumulator owns a BufferPool of buffer.memory bytes. The pool keeps a free list of pre-allocated ByteBuffers of exactly batch.size bytes (default 16384) — that's the "Poolable size" in the error message. Allocations of batch.size are cheap free-list pops; records bigger than batch.size get one-off non-pooled allocations, which fragment the pool and make exhaustion more likely with large records.

The append path: doSend() serializes the record, picks a partition, and calls RecordAccumulator.append(), which either finds an open batch for that partition with room, or allocates a new buffer from the pool. When the pool has no free memory, the calling thread parks on a condition queue inside BufferPool.allocate() — FIFO, so one stuck thread holds up the ones behind it — with a deadline of max.block.ms. Note this budget is shared: metadata fetch and buffer allocation both draw from the same max.block.ms per send() call.

Memory is only returned when the sender thread completes a batch: broker acks it (or fails it terminally), the batch's buffer is deallocated back to the pool, and waiting threads are signalled. So the drain rate is bounded by broker throughput × your ack settings. With acks=all and a struggling ISR, batch completion latency explodes, deallocation stalls, and the pool pins at Available memory: 0 bytes — exactly what the exception prints.

Two multipliers make it worse:

• Head-of-line partition stalls. One partition with a leaderless or slow broker keeps its in-flight batches unacked. Those buffers stay allocated while healthy partitions' traffic competes for what's left.
• Poor batching. Tiny records into many partitions with linger.ms=0 produce many under-filled batch.size buffers. You exhaust the pool at a fraction of its theoretical byte capacity.

One thing this exception is not: a broker problem per se. The broker never sees these records. Anything you don't catch here is silent data loss on the application side — send() threw before Kafka ever had the record.

5. The Fix

First decide which failure you have — a drain problem (broker slow/unavailable) or an inflow problem (producer genuinely outruns a healthy cluster). Check record-queue-time-avg and broker health: if the cluster was degraded, fix that; no client tuning outruns a paused broker.

For the inflow case, tune the producer:

 # producer.properties
 bootstrap.servers=localhost:9092
-# buffer.memory=33554432        # default 32 MB
-# max.block.ms=60000            # default 60 s
-# linger.ms=0                   # default
-# compression.type=none         # default
+buffer.memory=134217728         # 128 MB: size for burst absorption (see formula below)
+max.block.ms=10000              # fail fast; 60 s of blocked request threads helps nobody
+linger.ms=10                    # fill batches instead of shipping fragments
+batch.size=65536                # bigger batches = fewer pool allocations, better drain
+compression.type=lz4            # fewer bytes to drain per record

Sizing buffer.memory isn't guesswork:

buffer.memory ≥ peak_produce_rate_bytes_per_sec × p99_batch_completion_secs × safety_factor(2)

E.g. 20 MB/s peak with p99 batch completion of 2 s during failovers → 20 × 2 × 2 = 80 MB. The default 32 MB assumes a healthy cluster and modest throughput; a burst during a leader election blows through it fast.

Which knob, when:

• POC / low throughput: leave defaults; if you hit this locally, your broker container is down or unreachable — fix that, not buffer.memory.
• High throughput, bursty: raise buffer.memory, add linger.ms=5–20 + lz4/zstd. Batching is the highest-leverage change — it raises drain rate and lowers allocation rate.
• Latency-sensitive APIs: keep max.block.ms low (2–10 s) and treat the exception as backpressure (next section). A 60 s block inside an HTTP handler is an outage of its own — you've converted a Kafka stall into a thread-pool exhaustion in your web tier.

And always handle the throw — send() throwing synchronously means the callback never fires:

try {
    producer.send(record, (md, ex) -> {
        if (ex != null) handleAsyncFailure(record, ex);
    });
} catch (BufferExhaustedException e) {   // extends TimeoutException
    handleBackpressure(record, e);       // shed, buffer to disk, or 429 upstream
}

6. Best Practices & The Better Design

The class-level fix is to treat producer memory as the bounded queue it is, and propagate backpressure instead of discovering it via exceptions.

Bound your inflow explicitly. The clean pattern is a semaphore sized below the pool's capacity, released on batch completion:

public class BackpressureProducer implements AutoCloseable {
    private final Producer<String, String> producer;
    private final Semaphore inFlight = new Semaphore(10_000); // tune to buffer.memory / avg record size

    public BackpressureProducer(Producer<String, String> producer) {
        this.producer = producer;
    }

    /** Returns false instead of blocking the caller for max.block.ms. */
    public boolean trySend(ProducerRecord<String, String> record) {
        if (!inFlight.tryAcquire()) {
            return false; // caller decides: 429, spill to outbox, drop with metric
        }
        try {
            producer.send(record, (md, ex) -> {
                inFlight.release();
                if (ex != null) failureHandler(record, ex);
            });
            return true;
        } catch (RuntimeException e) {
            inFlight.release();
            throw e;
        }
    }

    private void failureHandler(ProducerRecord<String, String> r, Exception ex) { /* DLQ/log */ }

    @Override public void close() { producer.close(); }
}

Now saturation surfaces as a false return you can act on — reject upstream, spill to an outbox table, or degrade — instead of 200 request threads parked inside BufferPool.allocate().

If the data must not be lost, front it with an outbox. Write to a local durable store (DB outbox, Debezium/poller relay) and let the relay absorb Kafka slowdowns. Producer buffer memory is a shock absorber, not a durability mechanism.

Spring Kafka equivalent — sane defaults in one place:

spring:
  kafka:
    producer:
      compression-type: lz4
      batch-size: 65536
      buffer-memory: 134217728
      properties:
        linger.ms: 10
        max.block.ms: 10000
        delivery.timeout.ms: 120000

KafkaTemplate.send() returns a CompletableFuture, but the max.block.ms block and the BufferExhaustedException still happen synchronously inside the calling thread — the future doesn't save your HTTP handler from parking.

7. How to Prevent It Long-Term

Monitor the pool, not just the errors. The producer exposes JMX under kafka.producer:type=producer-metrics,client-id=*:

• buffer-available-bytes — alert when it trends toward 0 under steady load.
• bufferpool-wait-ratio / bufferpool-wait-time-total — fraction of time appenders spend blocked on allocation. Anything consistently above ~0.1 means you're living on the edge; exhaustion is one broker hiccup away.
• record-queue-time-avg — rising queue time = drain rate falling behind; this is your leading indicator, usually minutes ahead of the first exception.
• batch-size-avg vs batch.size — chronically under-filled batches point at linger.ms/keying problems.

Load-test the failure, not just the throughput. Your perf test should include a broker stall (docker pause, or kill the partition leader) at peak produce rate and assert the application's behavior: does it shed load in a controlled way, or does every worker thread pile up in send()? The repro in section 2 is trivially scriptable into CI.

Team config conventions. Ban ad-hoc producer configs; ship a golden producer config (compression, linger, buffer sizing formula, max.block.ms ≤ your caller's timeout) as a shared library or Spring auto-config. Most buffer exhaustion incidents I've debugged trace back to one service with linger.ms=0, default buffer, and 3× everyone else's record rate.

Capacity planning. Buffer memory is per producer instance and off-heap-ish in accounting terms (it's heap, but outside your normal object churn) — 128 MB × 20 producer instances in one JVM is real memory. Budget it, and prefer fewer, shared producer instances per JVM; KafkaProducer is thread-safe and batches better when shared.

8. Key Takeaways

• BufferExhaustedException means send() couldn't get accumulator memory within max.block.ms: your app produced faster than the sender thread could drain to the brokers. It extends TimeoutException (since kafka-clients 2.8) — don't confuse it with metadata or batch-expiry timeouts.
• The record never reached Kafka. Catch it around send() — the callback will not fire for this failure.
• Diagnose direction first: slow drain (broker/ISR degradation — fix the cluster) vs. fast inflow (tune batching: linger.ms, batch.size, compression, then buffer.memory).
• Size buffer.memory ≥ peak byte rate × p99 batch-completion time × 2. The 32 MB default does not survive a leader failover at high throughput.
• Watch buffer-available-bytes and bufferpool-wait-ratio; a rising record-queue-time-avg predicts exhaustion before it happens. Bound inflow with a semaphore or outbox instead of letting threads park in BufferPool.allocate().