Kafka RecordTooLargeException: The 1MB Limit and How to Fix It

1. The Error

Two variants, and telling them apart is the first diagnostic step. Client-side rejection — the record never leaves your JVM:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 2097252 bytes
when serialized which is larger than 1048576, which is the value of the
max.request.size configuration.

Broker-side rejection — the record left the producer, the broker refused it:

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message
larger than the max message size the server will accept.

In Spring Kafka you'll usually see it wrapped:

org.springframework.kafka.KafkaException: Send failed;
nested exception is org.apache.kafka.common.errors.RecordTooLargeException: ...

Typical setup where this bites: kafka-clients 3.x–4.x, Spring Kafka 3.x, any broker from 2.x through 4.x — the defaults haven't moved in a decade. It shows up the first time someone publishes a full document scan, a batch export, or an Avro record with an embedded base64 blob. Works fine in the POC with toy payloads, explodes the day real data arrives.

The two variants have different fixes. Read the message text carefully: "the value of the max.request.size configuration" means the check failed in KafkaProducer.send(). "The max message size the server will accept" means the broker returned error code MESSAGE_TOO_LARGE — your producer config is fine, the broker/topic limit isn't.

2. How to Reproduce It (step-by-step)

Minimal KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:19092 --create --topic docs --partitions 3

Producer that sends a 2 MB payload (kafka-clients pinned):

// build.gradle: implementation 'org.apache.kafka:kafka-clients:3.9.1'
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);

try (var producer = new KafkaProducer<String, byte[]>(props)) {
    byte[] payload = new byte[2 * 1024 * 1024]; // 2 MB
    producer.send(new ProducerRecord<>("docs", "doc-1", payload),
        (md, ex) -> { if (ex != null) ex.printStackTrace(); });
    producer.flush();
}

This throws the client-side variant synchronously from send() — before any network I/O.

To reproduce the broker-side variant, raise the producer limit but leave the broker at its default:

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 5 * 1024 * 1024);

Now send() accepts the record, the batch goes over the wire, and the broker responds with MESSAGE_TOO_LARGE. Note: this error is not retriable — the record fails immediately regardless of retries, and the callback/future gets the exception.

You can trigger the broker-side variant from the CLI too:

head -c 2000000 /dev/urandom | base64 -w0 > big.txt
docker compose exec -T kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:19092 --topic docs \
  --producer-property max.request.size=5242880 < big.txt

3. Why It Happens — Surface Level

Kafka enforces a maximum record size at (at least) two independent checkpoints, and each has its own config:

The defaults are all "about 1 MB", which is why everything under a megabyte sails through and the first real payload blows up. Fixing only one checkpoint moves the failure to the next one — the classic sequence is: bump max.request.size, redeploy, get the broker-side variant, then discover message.max.bytes exists.

4. Why It Happens — Under the Hood

The producer check runs on the uncompressed, serialized size. Inside KafkaProducer.doSend(), after key/value serialization, ensureValidRecordSize() compares an upper-bound estimate of the record's batch size (serialized key + value + headers + record/batch overhead) against max.request.size and against buffer.memory. This happens before compression and before the record is appended to the accumulator. Consequence: enabling compression.type=zstd does not get you past the client-side check, even if the payload would compress to 50 KB. Plenty of engineers flip on compression, see the same exception, and conclude compression is broken — it never got a chance to run.

The broker check runs on the compressed batch. The broker validates the size of each record batch as it arrives in the produce request — after producer-side compression. So compression genuinely helps against message.max.bytes, just not against max.request.size. This asymmetry is the single most confusing part of the whole topic: a 3 MB JSON document with max.request.size=5242880 and compression.type=zstd may compress to 200 KB on the wire and be accepted by an untouched broker. The producer limit you raised, the broker limit you didn't have to.

The numbers aren't quite equal on purpose. message.max.bytes defaults to 1048588 — 1 MiB plus 12 bytes of record batch overhead — precisely so a producer at the default max.request.size can't produce a batch the broker rejects.

Also in the size-check family: max.request.size additionally caps the entire produce request (which can carry many batches for multiple partitions), and buffer.memory (default 32 MB) must exceed any single record or you get a differently-worded RecordTooLargeException about the total memory buffer.

What about consumers and replication? Since KIP-74 (Kafka 0.10.1), fetch.max.bytes and max.partition.fetch.bytes are soft limits: if the first record batch in the first non-empty partition of a fetch is larger than the limit, it's returned anyway so the consumer always makes progress. Same for replica.fetch.max.bytes on the replication path. So on any remotely modern cluster, oversized records won't strand consumers or stall ISR replication. Legacy note: on pre-0.10.1 consumers this was a hard limit and oversized records would wedge the consumer with its own RecordTooLargeException — if you're googling from a 0.9-era stack trace, that's your issue. On modern clusters, still align the fetch settings with your max message size for sane batching, but they won't cause hard failures.

5. The Fix

Decide the real maximum record size first (measure your P100 payload, add headroom — say 10 MB), then raise the limit at every checkpoint, ideally per-topic rather than cluster-wide.

Step 1 — topic level (preferred over cluster-wide):

/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:19092 \
  --alter --entity-type topics --entity-name docs \
  --add-config max.message.bytes=10485760

Step 2 — producer:

 props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 10485760);      // 10 MiB
+props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
+// large batches take longer end-to-end; give the delivery pipeline room
+props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 180000);

Spring Boot equivalent:

 spring:
   kafka:
     producer:
+      compression-type: zstd
+      properties:
+        max.request.size: 10485760
+        delivery.timeout.ms: 180000

Step 3 — only if you must raise it cluster-wide (all topics, so think twice):

 environment:
+  KAFKA_MESSAGE_MAX_BYTES: 10485760
+  KAFKA_REPLICA_FETCH_MAX_BYTES: 10485760

message.max.bytes is dynamically updatable per broker via kafka-configs.sh --entity-type brokers, no rolling restart needed on modern brokers.

Step 4 — consumers. Not strictly required post-KIP-74, but align them so fetches stay efficient:

max.partition.fetch.bytes=10485760
fetch.max.bytes=52428800

Which fix when:

• POC / local dev: bump max.request.size + topic max.message.bytes and move on.
• Production, payloads occasionally 1–10 MB: per-topic max.message.bytes on the specific topics, producer max.request.size raised only in the producing services, zstd compression on. Do not raise the cluster default for everyone.
• Production, payloads routinely >10 MB: stop. Don't config your way out. Use the claim-check pattern (next section). Big records inflate broker heap and page cache pressure, slow log recovery and replication, blow out consumer processing time (hello max.poll.interval.ms rebalances), and make tiered storage and quota math miserable.

6. Best Practices & The Better Design

Kafka is a log for events, not a blob store. The design that removes this entire class of error is the claim-check pattern: put the payload in object storage, put a reference in Kafka.

public void publishDocument(String docId, byte[] pdfBytes) {
    String key = "docs/%s/%s.pdf".formatted(LocalDate.now(), docId);

    s3.putObject(b -> b.bucket("doc-payloads").key(key)
            .contentType("application/pdf"),
        RequestBody.fromBytes(pdfBytes));

    // The Kafka record is now ~200 bytes, forever below any limit
    DocumentEvent event = new DocumentEvent(docId, "doc-payloads", key,
        pdfBytes.length, sha256(pdfBytes));
    kafkaTemplate.send("doc-events", docId, event);
}

Rules that make claim-check production-grade: write the blob before the Kafka send (consumers must never dereference a pointer to nothing); include size and checksum in the event so consumers can validate; align bucket lifecycle policy with topic retention (an event that outlives its blob is a poison reference); keep the event self-describing enough that most consumers don't need the blob at all.

If your payloads are only sometimes large (e.g., 99% under 512 KB), a hybrid works well: inline small payloads, claim-check large ones, one boolean field in the envelope saying which. You keep single-hop latency for the common case.

And regardless of design: turn on compression.type=zstd (or lz4 if CPU-bound) at the producer. It's the cheapest throughput win in Kafka, it just isn't a fix for the client-side size check.

7. How to Prevent It Long-Term

Guardrail at build time. Codify the max payload size in the producing service — validate before send() and emit a domain-specific error instead of letting RecordTooLargeException surface three layers up in a Spring callback.

Contract-level enforcement. If you run Schema Registry, keep blobs out of schemas entirely — a bytes field in an Avro schema is a size incident waiting to happen. Review for it in schema PRs.

Monitoring. Alert on the producer JMX metric record-error-rate (kafka.producer:type=producer-metrics) — size rejections land there. Track record-size-avg/record-size-max (producer) and MessagesInPerSec vs BytesInPerSec ratio per topic to spot payload inflation before it crosses the limit. A payload P99 trending toward your limit is a two-week warning; use it.

Config conventions. Keep max.request.size ≤ topic max.message.bytes ≤ replica.fetch.max.bytes as a documented invariant. Encode topic size limits in your IaC (Terraform kafka_topic configs, or Strimzi KafkaTopic CRs) so they're reviewable, not tribal knowledge from a 2 a.m. kafka-configs.sh session.

Load-test with real payload distributions. Synthetic 1 KB messages in perf tests are how 2 MB production payloads stay undiscovered. Replay a sampled production corpus (sizes included) in staging.

8. Key Takeaways

• RecordTooLargeException has two variants — read the message: "max.request.size configuration" = client-side, fix the producer; "server will accept" = broker-side, fix message.max.bytes (topic: max.message.bytes).
• The producer checks uncompressed serialized size; the broker checks the compressed batch. Compression helps only at the broker checkpoint.
• Broker-side MESSAGE_TOO_LARGE is not retriable — no amount of retries will get an oversized record in.
• Raise limits per-topic, keep max.request.size ≤ max.message.bytes ≤ replica.fetch.max.bytes, and remember fetch limits are soft since KIP-74 — modern consumers won't wedge.
• Payloads routinely over a few MB don't belong in Kafka. Claim-check: blob to object storage, reference in the event.