> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka RecordTooLargeException: The 1MB Limit and How to Fix It - URL: https://www.ggorantala.dev/kafka-recordtoolargeexception-fix/ - Published: 2025-12-10T14:19:00.000Z - Updated: 2026-08-29T08:25:57.000Z - Description: Hitting RecordTooLargeException in Kafka? Learn why the 1MB default bites on both producer and broker, and the exact configs to fix it properly. - Author: Gopi Gorantala - Tags: kafka-errors, recordtoolargeexception, kafka-producer, spring-kafka, kafka-config, Java, event-streaming 1. The Error Two variants, and telling them apart is the first diagnostic step. **Client-side rejection** — the record never leaves your JVM: ```text org.apache.kafka.common.errors.RecordTooLargeException: The message is 2097252 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration. ``` **Broker-side rejection** — the record left the producer, the broker refused it: ```text org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept. ``` In Spring Kafka you'll usually see it wrapped: ```text org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: ... ``` Typical setup where this bites: kafka-clients 3.x–4.x, Spring Kafka 3.x, any broker from 2.x through 4.x — the defaults haven't moved in a decade. It shows up the first time someone publishes a full document scan, a batch export, or an Avro record with an embedded base64 blob. Works fine in the POC with toy payloads, explodes the day real data arrives. The two variants have different fixes. Read the message text carefully: "the value of the max.request.size configuration" means the check failed in `KafkaProducer.send()`. "The max message size the server will accept" means the broker returned error code `MESSAGE_TOO_LARGE` — your producer config is fine, the broker/topic limit isn't. ## 2\. How to Reproduce It (step-by-step) Minimal KRaft cluster: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.1 ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ``` ```bash docker compose up -d docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:19092 --create --topic docs --partitions 3 ``` Producer that sends a 2 MB payload (kafka-clients pinned): ```java // build.gradle: implementation 'org.apache.kafka:kafka-clients:3.9.1' Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class); try (var producer = new KafkaProducer(props)) { byte[] payload = new byte[2 * 1024 * 1024]; // 2 MB producer.send(new ProducerRecord<>("docs", "doc-1", payload), (md, ex) -> { if (ex != null) ex.printStackTrace(); }); producer.flush(); } ``` This throws the **client-side** variant synchronously from `send()` — before any network I/O. To reproduce the **broker-side** variant, raise the producer limit but leave the broker at its default: ```java props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 5 * 1024 * 1024); ``` Now `send()` accepts the record, the batch goes over the wire, and the broker responds with `MESSAGE_TOO_LARGE`. Note: this error is **not retriable** — the record fails immediately regardless of `retries`, and the callback/future gets the exception. You can trigger the broker-side variant from the CLI too: ```bash head -c 2000000 /dev/urandom | base64 -w0 > big.txt docker compose exec -T kafka /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:19092 --topic docs \ --producer-property max.request.size=5242880 < big.txt ``` ## 3\. Why It Happens — Surface Level Kafka enforces a maximum record size at (at least) two independent checkpoints, and each has its own config: | Checkpoint | Config | Default | Scope | | ------------------- | ----------------- | --------------- | ------------ | | Producer, in send() | max.request.size | 1048576 (1 MiB) | per producer | | Broker, on produce | message.max.bytes | 1048588 | cluster-wide | | Topic, on produce | max.message.bytes | inherits broker | per topic | The defaults are all "about 1 MB", which is why everything under a megabyte sails through and the first real payload blows up. Fixing only one checkpoint moves the failure to the next one — the classic sequence is: bump `max.request.size`, redeploy, get the broker-side variant, then discover `message.max.bytes` exists. ## 4\. Why It Happens — Under the Hood **The producer check runs on the uncompressed, serialized size.** Inside `KafkaProducer.doSend()`, after key/value serialization, `ensureValidRecordSize()` compares an upper-bound estimate of the record's batch size (serialized key + value + headers + record/batch overhead) against `max.request.size` and against `buffer.memory`. This happens before compression and before the record is appended to the accumulator. Consequence: enabling `compression.type=zstd` does **not** get you past the client-side check, even if the payload would compress to 50 KB. Plenty of engineers flip on compression, see the same exception, and conclude compression is broken — it never got a chance to run. **The broker check runs on the compressed batch.** The broker validates the size of each *record batch* as it arrives in the produce request — after producer-side compression. So compression genuinely helps against `message.max.bytes`, just not against `max.request.size`. This asymmetry is the single most confusing part of the whole topic: a 3 MB JSON document with `max.request.size=5242880` and `compression.type=zstd` may compress to 200 KB on the wire and be accepted by an untouched broker. The producer limit you raised, the broker limit you didn't have to. **The numbers aren't quite equal on purpose.** `message.max.bytes` defaults to 1048588 — 1 MiB plus 12 bytes of record batch overhead — precisely so a producer at the default `max.request.size` can't produce a batch the broker rejects. **Also in the size-check family:** `max.request.size` additionally caps the entire produce *request* (which can carry many batches for multiple partitions), and `buffer.memory` (default 32 MB) must exceed any single record or you get a differently-worded `RecordTooLargeException` about the total memory buffer. **What about consumers and replication?** Since KIP-74 (Kafka 0.10.1), `fetch.max.bytes` and `max.partition.fetch.bytes` are soft limits: if the first record batch in the first non-empty partition of a fetch is larger than the limit, it's returned anyway so the consumer always makes progress. Same for `replica.fetch.max.bytes` on the replication path. So on any remotely modern cluster, oversized records won't strand consumers or stall ISR replication. Legacy note: on pre-0.10.1 consumers this was a hard limit and oversized records would wedge the consumer with its own `RecordTooLargeException` — if you're googling from a 0.9-era stack trace, that's your issue. On modern clusters, still align the fetch settings with your max message size for sane batching, but they won't cause hard failures. ## 5\. The Fix Decide the real maximum record size first (measure your P100 payload, add headroom — say 10 MB), then raise the limit at **every checkpoint**, ideally per-topic rather than cluster-wide. **Step 1 — topic level (preferred over cluster-wide):** ```bash /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:19092 \ --alter --entity-type topics --entity-name docs \ --add-config max.message.bytes=10485760 ``` **Step 2 — producer:** ```diff props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); +props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 10485760); // 10 MiB +props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd"); +// large batches take longer end-to-end; give the delivery pipeline room +props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 180000); ``` Spring Boot equivalent: ```diff spring: kafka: producer: + compression-type: zstd + properties: + max.request.size: 10485760 + delivery.timeout.ms: 180000 ``` **Step 3 — only if you must raise it cluster-wide** (all topics, so think twice): ```diff environment: + KAFKA_MESSAGE_MAX_BYTES: 10485760 + KAFKA_REPLICA_FETCH_MAX_BYTES: 10485760 ``` `message.max.bytes` is dynamically updatable per broker via `kafka-configs.sh --entity-type brokers`, no rolling restart needed on modern brokers. **Step 4 — consumers.** Not strictly required post-KIP-74, but align them so fetches stay efficient: ```properties max.partition.fetch.bytes=10485760 fetch.max.bytes=52428800 ``` **Which fix when:** - **POC / local dev:** bump `max.request.size` \+ topic `max.message.bytes` and move on. - **Production, payloads occasionally 1–10 MB:** per-topic `max.message.bytes` on the specific topics, producer `max.request.size` raised only in the producing services, zstd compression on. Do not raise the cluster default for everyone. - **Production, payloads routinely >10 MB:** stop. Don't config your way out. Use the claim-check pattern (next section). Big records inflate broker heap and page cache pressure, slow log recovery and replication, blow out consumer processing time (hello `max.poll.interval.ms` rebalances), and make tiered storage and quota math miserable. ## 6\. Best Practices & The Better Design Kafka is a log for events, not a blob store. The design that removes this entire class of error is the **claim-check pattern**: put the payload in object storage, put a reference in Kafka. ```java public void publishDocument(String docId, byte[] pdfBytes) { String key = "docs/%s/%s.pdf".formatted(LocalDate.now(), docId); s3.putObject(b -> b.bucket("doc-payloads").key(key) .contentType("application/pdf"), RequestBody.fromBytes(pdfBytes)); // The Kafka record is now ~200 bytes, forever below any limit DocumentEvent event = new DocumentEvent(docId, "doc-payloads", key, pdfBytes.length, sha256(pdfBytes)); kafkaTemplate.send("doc-events", docId, event); } ``` Rules that make claim-check production-grade: write the blob **before** the Kafka send (consumers must never dereference a pointer to nothing); include size and checksum in the event so consumers can validate; align bucket lifecycle policy with topic retention (an event that outlives its blob is a poison reference); keep the event self-describing enough that most consumers don't need the blob at all. If your payloads are only *sometimes* large (e.g., 99% under 512 KB), a hybrid works well: inline small payloads, claim-check large ones, one boolean field in the envelope saying which. You keep single-hop latency for the common case. And regardless of design: turn on `compression.type=zstd` (or `lz4` if CPU-bound) at the producer. It's the cheapest throughput win in Kafka, it just isn't a fix for the client-side size check. ## 7\. How to Prevent It Long-Term **Guardrail at build time.** Codify the max payload size in the producing service — validate before `send()` and emit a domain-specific error instead of letting `RecordTooLargeException` surface three layers up in a Spring callback. **Contract-level enforcement.** If you run Schema Registry, keep blobs out of schemas entirely — a `bytes` field in an Avro schema is a size incident waiting to happen. Review for it in schema PRs. **Monitoring.** Alert on the producer JMX metric `record-error-rate` (kafka.producer:type=producer-metrics) — size rejections land there. Track `record-size-avg`/`record-size-max` (producer) and `MessagesInPerSec` vs `BytesInPerSec` ratio per topic to spot payload inflation before it crosses the limit. A payload P99 trending toward your limit is a two-week warning; use it. **Config conventions.** Keep `max.request.size ≤ topic max.message.bytes ≤ replica.fetch.max.bytes` as a documented invariant. Encode topic size limits in your IaC (Terraform `kafka_topic` configs, or Strimzi `KafkaTopic` CRs) so they're reviewable, not tribal knowledge from a 2 a.m. `kafka-configs.sh` session. **Load-test with real payload distributions.** Synthetic 1 KB messages in perf tests are how 2 MB production payloads stay undiscovered. Replay a sampled production corpus (sizes included) in staging. ## 8\. Key Takeaways - `RecordTooLargeException` has two variants — read the message: "max.request.size configuration" = client-side, fix the producer; "server will accept" = broker-side, fix `message.max.bytes` (topic: `max.message.bytes`). - The producer checks **uncompressed** serialized size; the broker checks the **compressed** batch. Compression helps only at the broker checkpoint. - Broker-side `MESSAGE_TOO_LARGE` is not retriable — no amount of `retries` will get an oversized record in. - Raise limits per-topic, keep `max.request.size ≤ max.message.bytes ≤ replica.fetch.max.bytes`, and remember fetch limits are soft since KIP-74 — modern consumers won't wedge. - Payloads routinely over a few MB don't belong in Kafka. Claim-check: blob to object storage, reference in the event.