> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # RecordTooLargeException: Fixing Kafka's Message Size Limits - URL: https://www.ggorantala.dev/kafka-recordtoolargeexception-message-size-limits/ - Published: 2026-09-14T18:40:58.000Z - Updated: 2026-09-14T18:42:34.000Z - Description: RecordTooLargeException fires from two different places for two different reasons. Here's how to tell which one hit you, and the fix that actually holds. - Author: Gopi Gorantala - Tags: apache-kafka, kafka-errors, recordtoolargeexception, kafka-producer, message-max-bytes, max-request-size, kafka-broker-config ## What shows up in the logs You send a record that's a bit too big and get this back, either straight out of `producer.send()` or wrapped in an `ExecutionException` from the future: ```log org.apache.kafka.common.errors.RecordTooLargeException: The message is 1500028 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration. ``` Or, if you already "fixed" it once, this one instead: ```log org.apache.kafka.common.errors.RecordTooLargeException: The record batch size in the append to size-repro-0 is 3000038 bytes which exceeds the maximum configured value of 1048588). ``` Same exception class, same-sounding message, and I'd bet most people who hit the second one assume they're looking at the first one again with a bigger number. They're not. One of these never leaves your JVM. The other one is your broker, after a real network round trip, telling you no. Which one you're looking at changes what you fix, and fixing the wrong one is how you end up raising `max.request.size` twice and still failing. I went and read the throw sites for both of these in the 4.3.1 source rather than trust what six-year-old Stack Overflow answers say about them, because the two messages above come from genuinely different code paths with a details gap between them that nobody seems to write down. ## Local check or broker rejection? The fast way to tell them apart: how did the exception arrive? If it came out of `producer.send()` itself, synchronously, before the call even returns a `Future`, that's the client. `KafkaProducer` checks record size in `doSend()`, before the record ever touches the accumulator or a socket. No broker was involved. No bytes went anywhere. If it came out of `future.get()` as an `ExecutionException`, or through your `Callback`, after `send()` returned normally, that's the broker. The request made it onto the wire, got parsed, and was rejected during log append. The broker side has a second, quieter tell: it doesn't log anything. `ReplicaManager.appendToLocalLog` catches `RecordTooLargeException` in the same branch as `UnknownTopicOrPartitionException` and `CorruptRecordException`, with a comment saying flat out that these are expected, known outcomes and shouldn't count as broker failures. No `WARN`, no `ERROR`, nothing in `server.log`. Go looking there for a smoking gun and you'll find silence. That silence is itself the confirmation you want. What does move is the per-topic `BytesRejectedPerSec` JMX metric (`kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=`), which ticks up on every rejection whether or not anyone's watching the logs. ## Reproduce it on your laptop **a. What you need.** `apache/kafka:4.3.1` (Docker), JDK 17, `kafka-clients:4.3.1` on the classpath. One broker is enough. This never touches replication. **b. The broker.** The stock single-node KRaft compose from the Kafka repo, unmodified. That's the point: you don't need to misconfigure anything to hit this. ```yaml # docker/examples/docker-compose-files/single-node/plaintext, unmodified services: broker: image: apache/kafka:4.3.1 container_name: broker ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: PLAINTEXT://broker:19092,CONTROLLER://broker:19093,PLAINTEXT_HOST://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:19092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:19093 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LOG_DIRS: /tmp/kraft-combined-logs CLUSTER_ID: 4L6g3nShT-eMCtK--X86sw ``` **c. The topic.** Left at every default. `message.max.bytes` resolves to `1024*1024 + 12 = 1048588` bytes (`Records.LOG_OVERHEAD` is 12), and that's the ceiling neither of our sends will clear: ```bash docker compose up -d docker exec broker /opt/kafka/bin/kafka-topics.sh --create \ --topic size-repro --bootstrap-server localhost:9092 --partitions 1 ``` **d. The reproducer.** ```xml org.apache.kafka kafka-clients 4.3.1 org.slf4j slf4j-simple 1.7.36 ``` ```java package com.example.repro; import org.apache.kafka.clients.producer.*; import org.apache.kafka.common.errors.RecordTooLargeException; import org.apache.kafka.common.serialization.ByteArraySerializer; import org.apache.kafka.common.serialization.StringSerializer; import java.util.Properties; import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; public class RecordSizeRepro { public static void main(String[] args) throws Exception { String scenario = args.length > 0 ? args[0] : "client"; if ("client".equals(scenario)) runClientSideRejection(); else runBrokerSideRejection(); } // Step 1: defaults everywhere, 1.5 MB value. Rejected before any network call. private static void runClientSideRejection() { Properties props = baseProps(); try (KafkaProducer producer = new KafkaProducer<>(props)) { byte[] payload = new byte[1_500_000]; System.out.println("Calling send(), default max.request.size is 1048576..."); producer.send(new ProducerRecord<>("size-repro", "k1", payload)); System.out.println("unreachable"); } catch (RecordTooLargeException e) { System.out.println("Caught on the calling thread, no request was sent:"); System.out.println(e.getMessage()); } } // Step 2: raise the producer's own ceiling, leave the topic at its default. private static void runBrokerSideRejection() throws Exception { Properties props = baseProps(); props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 5_242_880); props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67_108_864L); try (KafkaProducer producer = new KafkaProducer<>(props)) { byte[] payload = new byte[3_000_000]; System.out.println("Calling send(), max.request.size raised to 5 MB, topic still at ~1 MB..."); Future f = producer.send(new ProducerRecord<>("size-repro", "k2", payload)); System.out.println("send() returned normally. Client-side check passed. Waiting on the broker..."); f.get(); System.out.println("unreachable"); } catch (ExecutionException e) { System.out.println("Caught from future.get(), after a real produce request:"); System.out.println(e.getCause()); } } private static Properties baseProps() { Properties p = new Properties(); p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); p.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 10_000); return p; } } ``` ```log Step 1: run scenario 1 $ java -cp target/repro-1.0.jar com.example.repro.RecordSizeRepro client Expected: prints the RecordTooLargeException referencing max.request.size, in under a second, no broker traffic. Step 2: run scenario 2 $ java -cp target/repro-1.0.jar com.example.repro.RecordSizeRepro broker Expected: "send() returned normally" prints first, then a couple hundred ms later the RecordTooLargeException referencing 1048588 comes back from future.get(). ``` **e. Confirmation signal.** Scenario 1's exception text names `max.request.size`. Scenario 2's names a byte count ending `)` with no matching `(`. That stray parenthesis is a genuine typo that's been sitting in this message since at least Kafka 2.8.2\. If you see `RecordBatchTooLargeException` instead, you've hit a different, unrelated ceiling (`segment.bytes`, not this one); that only happens if someone shrank the segment size far below the message size limit for testing and forgot to put it back. **f. Teardown.** `docker compose down -v`. The named volume holds the KRaft metadata log, and starting fresh against stale `meta.properties` produces an unrelated and much more confusing failure on next boot. ## The short answer Scenario 1: your record is bigger than `max.request.size`, and the Java producer catches that itself before wasting a network round trip. Scenario 2: your record cleared the producer's own limit but not the topic's, so the broker parsed the whole request and rejected it during log append. Both throw the same exception class because, structurally, they're both "you tried to write something bigger than the configured ceiling." The ceiling lives in a different place, that's all. ## Two throw sites, one exception name Here's the part that isn't in the client-side story at all: the broker doesn't have one check, it has two, and they're not equally likely to fire. `UnifiedLog.append()` calls `analyzeAndValidateRecords()` first, and that method checks every batch's size against `config().maxMessageSize()` before anything else happens: no compression, no offset assignment yet. This is the check that produced scenario 2's message above, and it's the one you'll hit nearly every time, because it runs on the request exactly as the producer sent it. There's a *second* check, later in the same `append()` call, guarded by `messageSizeMaybeChanged()`. That flag only comes back `true` if the broker had to rebuild the batch: down-converting it for an ancient client, or recompressing it because the topic's `compression.type` differs from what the producer sent. If the batch grows past the limit only after that rebuild, you get a second, differently-worded exception: "Message batch size is *N* bytes in append to*partition* ...". No, that missing space isn't a typo I made in this article; the source doesn't have one either. `UnifiedLog.java` really does concatenate `"in append to"` and `"partition "` with nothing between them. It's a cosmetic bug that's been there since the Scala `Log.scala` days, and I'd guess nobody's filed a Jira for it, because it's never the thing that actually costs anyone the debugging time. I only found it by diffing the exact string across five release tags, looking for wording drift. There wasn't any. That's worth knowing on its own: every Stack Overflow answer quoting either message is still accurate today, whichever year it was written in. The reason this matters: if you're staring at the *first* message and reach for "the broker recompressed my batch," you're chasing a red herring. Compression-triggered rejection is real, but it's rare. Normal produce traffic almost never touches that second path. I'd have bet money the two were the same check sharing one string constant, before I actually opened the file. Neither exception, by the way, is retriable. `MESSAGE_TOO_LARGE` (error code 10) is a permanent rejection on the wire, so `retries` and `delivery.timeout.ms` do nothing for either scenario. There's no transient condition to wait out. ## The fix Raising `max.request.size` alone doesn't fix scenario 2, it *causes* it. You're moving the rejection point from your own JVM to the broker, which means you now pay for a full network round trip before finding out the record still doesn't fit anywhere. All three levels have to agree, and they're three different config keys at three different levels: ```diff # Producer (client-side ceiling, checked before send) - max.request.size=1048576 # default + max.request.size=5242880 # 5 MB # Producer (must also cover the bigger request) - buffer.memory=33554432 # default, 32 MB + buffer.memory=67108864 # 64 MB, headroom above max.request.size # Topic (server-side ceiling, checked on append; set this, not the broker default) $ kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics --entity-name size-repro --alter \ --add-config max.message.bytes=5242880 ``` Prefer the topic-level `max.message.bytes` over the broker-level `message.max.bytes` cluster default, unless you genuinely want every topic on the cluster to accept bigger messages. The cluster-wide knob affects replication fetch sizes and page cache pressure for topics that never needed the headroom. On non-JVM clients built on librdkafka, the story's different in a way that matters. Its own `message.max.bytes` producer config is documented as advisory: "the producer is unable to reliably enforce a strict max message limit at produce time." A librdkafka producer can't guarantee the fast, synchronous, no-network rejection Java gives you in scenario 1\. The real enforcement is still the broker's, and it surfaces asynchronously through the delivery report as a local `MSG_SIZE_TOO_LARGE` error rather than an exception thrown from the send call. Named non-fixes, because all three show up in answers to this exact error. Raising `retries` or `request.timeout.ms` does nothing, since neither config is even consulted for a non-retriable error. Catching the exception and silently dropping the record buys you permanent, invisible data loss, tracked only by a metric (`BytesRejectedPerSec`) that nobody's alerting on. And raising `max.partition.fetch.bytes` on the *consumer* side is a leftover from before KIP-74 landed in 0.10.1\. Since then, the broker always returns at least one full record batch to guarantee progress even if it's over the limit, so this config can't produce a `RecordTooLargeException` on the consumer at all, and hasn't been able to for years. The Javadoc on `AsyncKafkaConsumer.poll()` still lists it as a checked exception; I think that's a stale `@throws` tag nobody's gotten around to deleting. In Spring Boot, there's no dedicated `spring.kafka.producer.max-request-size`. `KafkaProperties.Producer` has first-class fields for `batch-size` and `buffer-memory` but not this one. You set it through the generic passthrough: `spring.kafka.producer.properties.max.request.size=5242880`. ## Size limits, chosen on purpose The cleanest fix is usually not raising the limit at all. If you're regularly producing multi-megabyte records, ask why before you ask how big. Compression (`compression.type=lz4` or `zstd` on the producer) often gets a 3 MB JSON payload under the default 1 MB ceiling without touching any size config, and it's free bandwidth savings on every smaller record too. For records that are genuinely large by nature (images, file attachments, big blobs), the claim-check pattern, putting the blob in object storage and producing a small message that carries its reference, keeps Kafka doing what it's fast at instead of turning it into a file transfer protocol it was never built to be. If you do need the ceiling raised, raise it deliberately and narrowly: on the one topic that needs it, not the broker default, and document the number next to the producer config so the next person doesn't have to reverse-engineer why `max.request.size` is set to a strange value. ## What to alert on Alert on the per-topic `BytesRejectedPerSec` JMX metric (`kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=`) being non-zero over any window. Since `server.log` stays silent, this metric is the only broker-side signal you get for free, and any non-zero reading means something is actively losing data right now. On the producer, `record-size-max` (`kafka.producer:type=producer-metrics,client-id=`) tells you the largest record you've actually sent recently, so you can watch it trend toward your configured ceiling before it gets there, not after. The client-side rejection in scenario 1 doesn't move `record-error-rate` at all, because the record never reached the accumulator where that metric gets recorded. The exception itself is your only signal for that path, which is a good argument for never calling `producer.send()` without either a callback or checking the returned future. On versions: both throw sites and both message strings are unchanged from 2.8.2 through 4.3.1\. I diffed the source across five tags spanning that range and found byte-for-byte identical wording, missing space and stray parenthesis included, surviving the Scala-to-Java rewrite of the storage layer along the way. There's no "safe version" to upgrade to here and no version where this behaves differently. The one thing that *has* moved over that span is the consumer side. If you're running anything from before 0.10.1, the consumer-side variant of this exception was real and `max.partition.fetch.bytes` was the actual fix. On anything you're likely running today, it isn't. ## Five things to keep in your head - `RecordTooLargeException` from `producer.send()` directly means the client rejected it, no network trip happened, and the number in the message is `max.request.size` or `buffer.memory`. - The same exception from a `Future` or callback means the broker rejected it after a real request, and the number is the topic's `max.message.bytes`, which defaults to roughly 1 MB plus 12 bytes of overhead. - The broker logs nothing for either case. The confirmation is the `BytesRejectedPerSec` metric, not `server.log`. - Fix all three levels together (producer `max.request.size`, `buffer.memory`, topic `max.message.bytes`) or you've only relocated the failure, not fixed it. - Neither error is retriable. `MESSAGE_TOO_LARGE` is permanent, so `retries` and `delivery.timeout.ms` don't apply here.