> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka BufferExhaustedException: Fix Failed to Allocate Memory - URL: https://www.ggorantala.dev/kafka-bufferexhaustedexception-buffer-memory/ - Published: 2026-02-08T13:59:00.000Z - Updated: 2026-08-29T08:18:58.000Z - Description: Kafka producer throws BufferExhaustedException: failed to allocate memory within max blocking time? Here's why buffer.memory fills up and the exact fix. - Author: Gopi Gorantala - Tags: kafka-errors, bufferexhaustedexception, kafka-producer, buffer-memory, max-block-ms, Java, spring-kafka ## 1\. The Error Your producer thread — often *every* producer thread at once — starts throwing this out of `send()`: ```log org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 16384 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 16384 bytes at org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:132) at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:295) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1054) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:947) ``` On kafka-clients older than 2.8 the same failure surfaces as a plain `TimeoutException` with the identical `Failed to allocate memory within the configured max blocking time` message. Since 2.8, `BufferExhaustedException` extends `TimeoutException`, so old `catch (TimeoutException)` blocks still trigger — which is exactly why many teams mistake this for the *other* producer timeout. **Do not confuse it with these two:** - `TimeoutException: Topic X not present in metadata after 60000 ms` — metadata wait, also gated by `max.block.ms`, usually connectivity/listeners. - `TimeoutException: Expiring N record(s) for topic-0: 120000 ms has passed since batch creation` — batch already accepted, expired later under `delivery.timeout.ms`. `BufferExhaustedException` is earlier in the pipeline than both: `send()` couldn't even hand the record to the accumulator because the producer's `buffer.memory` pool is full and stayed full for `max.block.ms`. Typical setup where it bites: kafka-clients 3.x (any), Spring Boot 3.3.x with spring-kafka 3.2.x, high-throughput batch jobs or Kafka-connected APIs under burst load, and — the classic — a broker that slowed down or lost ISR while the app kept producing at full rate. ## 2\. How to Reproduce It Single KRaft broker: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.8.0 container_name: kafka ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:19092,CONTROLLER://0.0.0.0:9093,EXTERNAL://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ``` ```bash docker compose up -d docker exec kafka /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:19092 --create --topic orders --partitions 3 ``` Producer with a deliberately small pool and short block time (kafka-clients 3.8.0): ```java import org.apache.kafka.clients.producer.*; import java.util.Properties; public class BufferExhaustionRepro { public static void main(String[] args) { Properties p = new Properties(); p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer"); p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer"); p.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 1_048_576); // 1 MB, default 32 MB p.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 2_000); // default 60000 String payload = "x".repeat(1024); // 1 KB records try (Producer producer = new KafkaProducer<>(p)) { for (int i = 0; i < 1_000_000; i++) { producer.send(new ProducerRecord<>("orders", "k" + i, payload)); } } } } ``` Start it, then freeze the broker so the sender thread can't drain the accumulator: ```bash docker pause kafka ``` Within a few seconds the 1 MB pool is full, `send()` blocks, and after 2 s you get `BufferExhaustedException`. `docker unpause kafka` and it recovers. That's the whole failure mode in miniature: **inflow rate exceeded drain rate for longer than the buffer could absorb.** In production nobody sets `buffer.memory` to 1 MB — the same math just plays out at 32 MB with a traffic spike, a slow ISR follower, a leader failover, or a `min.insync.replicas` stall doing the job of `docker pause`. ## 3\. Why It Happens — Surface Level `KafkaProducer.send()` is asynchronous, but it isn't unbounded. Every serialized record must get space in a shared, fixed-size memory pool (`buffer.memory`, default 32 MB) before `send()` returns. A background sender thread drains that pool to the brokers. If your application appends faster than the sender can ship — because of a burst, a slow or unavailable broker, or too little batching — the pool fills. `send()` then blocks up to `max.block.ms` (default 60 s) waiting for a batch to complete and release memory. If nothing frees up in time, you get `BufferExhaustedException`. It's backpressure, surfaced as an exception. The producer is telling you the downstream can't keep up with the upstream, and its shock absorber is full. ## 4\. Why It Happens — Under the Hood Inside the producer, `RecordAccumulator` owns a `BufferPool` of `buffer.memory` bytes. The pool keeps a free list of pre-allocated `ByteBuffer`s of exactly `batch.size` bytes (default 16384) — that's the "Poolable size" in the error message. Allocations of `batch.size` are cheap free-list pops; records bigger than `batch.size` get one-off non-pooled allocations, which fragment the pool and make exhaustion *more* likely with large records. The append path: `doSend()` serializes the record, picks a partition, and calls `RecordAccumulator.append()`, which either finds an open batch for that partition with room, or allocates a new buffer from the pool. When the pool has no free memory, the calling thread parks on a condition queue inside `BufferPool.allocate()` — FIFO, so one stuck thread holds up the ones behind it — with a deadline of `max.block.ms`. Note this budget is *shared*: metadata fetch and buffer allocation both draw from the same `max.block.ms` per `send()` call. Memory is only returned when the sender thread completes a batch: broker acks it (or fails it terminally), the batch's buffer is deallocated back to the pool, and waiting threads are signalled. So the drain rate is bounded by broker throughput × your ack settings. With `acks=all` and a struggling ISR, batch completion latency explodes, deallocation stalls, and the pool pins at `Available memory: 0 bytes` — exactly what the exception prints. Two multipliers make it worse: - **Head-of-line partition stalls.** One partition with a leaderless or slow broker keeps its in-flight batches unacked. Those buffers stay allocated while healthy partitions' traffic competes for what's left. - **Poor batching.** Tiny records into many partitions with `linger.ms=0` produce many under-filled `batch.size` buffers. You exhaust the pool at a fraction of its theoretical byte capacity. One thing this exception is *not*: a broker problem per se. The broker never sees these records. Anything you don't catch here is silent data loss on the application side — `send()` threw before Kafka ever had the record. ## 5\. The Fix First decide which failure you have — a **drain problem** (broker slow/unavailable) or an **inflow problem** (producer genuinely outruns a healthy cluster). Check `record-queue-time-avg` and broker health: if the cluster was degraded, fix that; no client tuning outruns a paused broker. For the inflow case, tune the producer: ```diff # producer.properties bootstrap.servers=localhost:9092 -# buffer.memory=33554432 # default 32 MB -# max.block.ms=60000 # default 60 s -# linger.ms=0 # default -# compression.type=none # default +buffer.memory=134217728 # 128 MB: size for burst absorption (see formula below) +max.block.ms=10000 # fail fast; 60 s of blocked request threads helps nobody +linger.ms=10 # fill batches instead of shipping fragments +batch.size=65536 # bigger batches = fewer pool allocations, better drain +compression.type=lz4 # fewer bytes to drain per record ``` Sizing `buffer.memory` isn't guesswork: ```text buffer.memory ≥ peak_produce_rate_bytes_per_sec × p99_batch_completion_secs × safety_factor(2) ``` E.g. 20 MB/s peak with p99 batch completion of 2 s during failovers → 20 × 2 × 2 = 80 MB. The default 32 MB assumes a healthy cluster and modest throughput; a burst during a leader election blows through it fast. Which knob, when: - **POC / low throughput:** leave defaults; if you hit this locally, your broker container is down or unreachable — fix that, not `buffer.memory`. - **High throughput, bursty:** raise `buffer.memory`, add `linger.ms=5–20` \+ lz4/zstd. Batching is the highest-leverage change — it raises drain rate *and* lowers allocation rate. - **Latency-sensitive APIs:** keep `max.block.ms` low (2–10 s) and treat the exception as backpressure (next section). A 60 s block inside an HTTP handler is an outage of its own — you've converted a Kafka stall into a thread-pool exhaustion in your web tier. And always handle the throw — `send()` throwing synchronously means the callback never fires: ```java try { producer.send(record, (md, ex) -> { if (ex != null) handleAsyncFailure(record, ex); }); } catch (BufferExhaustedException e) { // extends TimeoutException handleBackpressure(record, e); // shed, buffer to disk, or 429 upstream } ``` ## 6\. Best Practices & The Better Design The class-level fix is to treat producer memory as the bounded queue it is, and propagate backpressure instead of discovering it via exceptions. **Bound your inflow explicitly.** The clean pattern is a semaphore sized below the pool's capacity, released on batch completion: ```java public class BackpressureProducer implements AutoCloseable { private final Producer producer; private final Semaphore inFlight = new Semaphore(10_000); // tune to buffer.memory / avg record size public BackpressureProducer(Producer producer) { this.producer = producer; } /** Returns false instead of blocking the caller for max.block.ms. */ public boolean trySend(ProducerRecord record) { if (!inFlight.tryAcquire()) { return false; // caller decides: 429, spill to outbox, drop with metric } try { producer.send(record, (md, ex) -> { inFlight.release(); if (ex != null) failureHandler(record, ex); }); return true; } catch (RuntimeException e) { inFlight.release(); throw e; } } private void failureHandler(ProducerRecord r, Exception ex) { /* DLQ/log */ } @Override public void close() { producer.close(); } } ``` Now saturation surfaces as a `false` return you can act on — reject upstream, spill to an outbox table, or degrade — instead of 200 request threads parked inside `BufferPool.allocate()`. **If the data must not be lost, front it with an outbox.** Write to a local durable store (DB outbox, Debezium/poller relay) and let the relay absorb Kafka slowdowns. Producer buffer memory is a shock absorber, not a durability mechanism. **Spring Kafka equivalent** — sane defaults in one place: ```yaml spring: kafka: producer: compression-type: lz4 batch-size: 65536 buffer-memory: 134217728 properties: linger.ms: 10 max.block.ms: 10000 delivery.timeout.ms: 120000 ``` `KafkaTemplate.send()` returns a `CompletableFuture`, but the `max.block.ms` block and the `BufferExhaustedException` still happen synchronously inside the calling thread — the future doesn't save your HTTP handler from parking. ## 7\. How to Prevent It Long-Term **Monitor the pool, not just the errors.** The producer exposes JMX under `kafka.producer:type=producer-metrics,client-id=*`: - `buffer-available-bytes` — alert when it trends toward 0 under steady load. - `bufferpool-wait-ratio` / `bufferpool-wait-time-total` — fraction of time appenders spend blocked on allocation. Anything consistently above \~0.1 means you're living on the edge; exhaustion is one broker hiccup away. - `record-queue-time-avg` — rising queue time = drain rate falling behind; this is your leading indicator, usually minutes ahead of the first exception. - `batch-size-avg` vs `batch.size` — chronically under-filled batches point at `linger.ms`/keying problems. **Load-test the failure, not just the throughput.** Your perf test should include a broker stall (`docker pause`, or kill the partition leader) at peak produce rate and assert the application's behavior: does it shed load in a controlled way, or does every worker thread pile up in `send()`? The repro in section 2 is trivially scriptable into CI. **Team config conventions.** Ban ad-hoc producer configs; ship a golden producer config (compression, linger, buffer sizing formula, `max.block.ms` ≤ your caller's timeout) as a shared library or Spring auto-config. Most buffer exhaustion incidents I've debugged trace back to one service with `linger.ms=0`, default buffer, and 3× everyone else's record rate. **Capacity planning.** Buffer memory is per producer instance and off-heap-ish in accounting terms (it's heap, but outside your normal object churn) — 128 MB × 20 producer instances in one JVM is real memory. Budget it, and prefer fewer, shared producer instances per JVM; `KafkaProducer` is thread-safe and batches better when shared. ## 8\. Key Takeaways - `BufferExhaustedException` means `send()` couldn't get accumulator memory within `max.block.ms`: your app produced faster than the sender thread could drain to the brokers. It extends `TimeoutException` (since kafka-clients 2.8) — don't confuse it with metadata or batch-expiry timeouts. - The record never reached Kafka. Catch it around `send()` — the callback will not fire for this failure. - Diagnose direction first: slow drain (broker/ISR degradation — fix the cluster) vs. fast inflow (tune batching: `linger.ms`, `batch.size`, compression, then `buffer.memory`). - Size `buffer.memory` ≥ peak byte rate × p99 batch-completion time × 2\. The 32 MB default does not survive a leader failover at high throughput. - Watch `buffer-available-bytes` and `bufferpool-wait-ratio`; a rising `record-queue-time-avg` predicts exhaustion before it happens. Bound inflow with a semaphore or outbox instead of letting threads park in `BufferPool.allocate()`.