Kafka CorruptRecordException: Record Is Corrupt — Fix It

Your consumer was running fine for weeks, then the whole group wedges on one partition and every instance throws the same stack trace in a tight loop. Lag climbs, nothing commits, restarts don't help — because the poison isn't in your code, it's on disk. This is the CRC integrity check firing, and unlike a poison pill you can't just deserialize your way around it.

This is a different animal from a RecordDeserializationException (bad payload, valid bytes) or a RecordTooLargeException (size policy). CorruptRecordException means the bytes on the wire or on disk don't match the checksum Kafka stored when it wrote them. Something flipped a bit between write and read.

1. The Error

The exact line you pasted into search, thrown out of poll():

org.apache.kafka.common.KafkaException: Received exception when fetching the next record from corrupt-demo-0. If needed, please seek past the record to continue consumption.
	at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:326)
	at org.apache.kafka.clients.consumer.internals.FetchCollector.fetchRecords(FetchCollector.java:168)
	...
Caused by: org.apache.kafka.common.errors.CorruptRecordException: Record for partition corrupt-demo-0 at offset 2911287689 is invalid, cause: Record is corrupt (stored crc = 3580880396, computed crc = 1701403171)

Broker-side, the same class shows up in server.log on the produce path or during log recovery:

[2026-07-12 09:14:22,317] ERROR Found invalid messages during fetch for partition corrupt-demo-0 offset 2911287689 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.CorruptRecordException: Record is corrupt (stored crc = 3580880396, computed crc = 1701403171)

Facts about this exception:

• org.apache.kafka.common.errors.CorruptRecordException, protocol error CORRUPT_MESSAGE, error code 2.
• It extends RetriableException. That is the trap: the client thinks a retry might help, so a fetch that hits persistently-corrupt bytes on disk retries forever, never advancing.
• Consumer-side it surfaces wrapped in a KafkaException from poll() — note the message "please seek past the record to continue consumption." That sentence is the whole recovery strategy in one line.

Applies to every modern line: kafka-clients / broker 0.11 through 4.0, KRaft or ZooKeeper, any language client that validates CRCs. The setup where it bites hardest is a single-replica topic (RF=1) — no clean copy to recover from.

2. How to Reproduce It

You can force this deterministically by corrupting a byte in a log segment. Single-broker KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

Create the topic and write a few records:

docker compose up -d

docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic corrupt-demo --partitions 1 --replication-factor 1

for i in $(seq 1 20); do echo "order-$i"; done | \
docker exec -i kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic corrupt-demo

Now flip a byte inside the segment. The record batch header near the front holds the CRC; corrupt a byte in the record body so the stored CRC no longer matches:

# Locate the segment
docker exec kafka ls -l /var/lib/kafka/data/corrupt-demo-0/

# Overwrite one byte at offset 120 with 0xFF, in place (do NOT truncate)
docker exec kafka bash -c \
  "printf '\xFF' | dd of=/var/lib/kafka/data/corrupt-demo-0/00000000000000000000.log \
   bs=1 seek=120 count=1 conv=notrunc"

# Drop page cache so the broker re-reads from disk
docker restart kafka

Consume from the beginning — the CRC check fires:

docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic corrupt-demo --from-beginning
# ... ERROR Error processing message, terminating consumer process:
# org.apache.kafka.common.errors.CorruptRecordException: Record is corrupt (stored crc = ...)

Confirm which record is bad without a consumer, using the log dumper — this is your production forensics tool:

docker exec kafka /opt/kafka/bin/kafka-dump-log.sh \
  --files /var/lib/kafka/data/corrupt-demo-0/00000000000000000000.log \
  --deep-iteration
# ... isvalid: false ... at the corrupted batch

Environment-specific triggers you'll see in the wild instead of dd: unflushed page cache lost on a hard power cut, a failing disk sector, non-ECC RAM bit flips, a truncated segment from a full disk mid-write, or a buggy/hand-rolled producer emitting malformed batches.

3. Why It Happens — Surface Level

Kafka stamps a CRC checksum onto every record batch when it's produced. On read, the consumer (with check.crcs=true, the default) recomputes the checksum over the bytes it received and compares. Mismatch → CorruptRecordException. The number stored on disk (stored crc) and the number computed from the current bytes (computed crc) disagree, so at least one bit changed since the write.

It is not your serializer, your schema, or your payload semantics. The batch is byte-damaged. Kafka refuses to hand you data it can't prove is intact.

4. Why It Happens — Under the Hood

Since message format v2 (magic byte 2, Kafka 0.11+), records are grouped into a RecordBatch and the checksum is CRC-32C (Castagnoli) — the hardware-accelerated variant, computed over the batch bytes after the CRC field (from the attributes field through the end of the batch). Format v0/v1 used CRC-32 (IEEE) per-record; both are long gone by 4.0, but the principle is identical: a checksum that covers the record bytes and travels with them.

The check runs in three places:

1. Produce path (broker). LogValidator validates incoming batches on append. In-flight corruption — a bit flipped by a flaky NIC, a bad switch, cosmic-ray RAM — is caught here and the broker returns CORRUPT_MESSAGE; the producer retries and usually succeeds on a clean resend. This is why network corruption rarely reaches disk.
2. Fetch path (consumer). With check.crcs=true, CompletedFetch.fetchRecords() re-validates each batch as it iterates. This catches corruption that happened at rest — the bytes were fine when written, then the disk sector rotted, or the OS page cache was lost uncleanly and the segment on disk is torn.
3. Log recovery (broker startup). After an unclean shutdown the broker scans segments; corrupt batches show up as recovery errors.

The reason a corrupt on-disk record wedges a consumer forever is the RetriableException inheritance. The fetcher gets bytes, validation fails, it treats the failure as transient and re-fetches the same offset — which returns the same corrupt bytes. There is no forward progress because retrying a deterministic on-disk corruption is pointless. The exception message spells out the only real fix: seek past it.

One nasty edge worth knowing (KAFKA-8722): during log compaction the cleaner copies retained records into new segments. When an offset isn't assigned "in place" (inPlaceAssignment=false), older versions skipped re-validating the CRC on the copied data, so a corrupt-but-retained record could be silently rewritten. If your corruption appears after compaction on a topic that was previously clean, suspect this class of bug and check your broker version.

Distinct cousin — InvalidRecordException. Don't confuse CorruptRecordException (code 2, bit-level integrity failure) with InvalidRecordException (INVALID_RECORD, code 87, KIP-467, Kafka 2.4+). The latter is the broker rejecting a well-formed-but-illegal record on produce: a null key to a compacted topic, non-monotonic offsets, a bad magic byte, an out-of-range timestamp. Same "your record is bad" vibe, completely different cause — InvalidRecordException is a policy/format rejection at write time; CorruptRecordException is checksum failure from actual data damage.

5. The Fix

There are two fixes and they're not interchangeable. First get the consumer moving again; then heal the data.

Fix A — Get consumers past the corrupt offset (the immediate unblock)

You cannot repair the bytes from the client, so skip them. Find the bad offset from the stack trace or kafka-dump-log.sh, then reset the group forward:

# Skip exactly the corrupt offset (jump to the next one)
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group orders-service \
  --topic corrupt-demo:0 \
  --reset-offsets --to-offset 2911287690 --execute

In application code, handle it deliberately rather than crash-looping. Before — the naive loop that dies on the first corrupt batch:

// BEFORE: one corrupt record on disk kills the consumer forever
while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> r : records) {
        process(r);
    }
    consumer.commitSync();
}

After — catch the wrapped exception, seek past the offender, keep going:

// AFTER: skip the corrupt offset and continue
while (running) {
    try {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> r : records) {
            process(r);
        }
        consumer.commitSync();
    } catch (KafkaException e) {
        if (e.getCause() instanceof CorruptRecordException) {
            // The exception does not expose the offset; track it from the last
            // successful position and advance one past the failing partition.
            for (TopicPartition tp : consumer.assignment()) {
                long pos = consumer.position(tp);
                log.error("Skipping corrupt record on {} at {}", tp, pos);
                consumer.seek(tp, pos + 1);
            }
        } else {
            throw e;
        }
    }
}

Skipping means data loss for those records. On a fintech-grade stream that is a decision, not a default — log it, alert on it, and reconcile from your source of truth.

Fix B — Heal the partition on the broker (RF ≥ 2 makes this trivial)

If the topic has replicas, the clean fix is to let Kafka re-replicate. Take the affected replica offline, delete the corrupt log directory for that partition on that broker, and let it re-fetch a clean copy from the leader:

# On the broker with the corrupt replica, with the broker stopped:
rm -rf /var/lib/kafka/data/corrupt-demo-0
# Restart the broker; it re-fetches the partition from the current leader.

Never hand-delete individual .log segments on a live broker or on the only replica — you'll strand offsets and indexes. With RF=1 you have no clean copy; your only options are skip-forward (Fix A) and accept the loss, or restore from an upstream/outbox.

Last-resort workaround — do NOT reach for this first

# Disables CRC validation on the consumer. Hides corruption, doesn't fix it.
check.crcs=false

Setting check.crcs=false makes the exception disappear by not looking. You'll happily deserialize garbage. Only ever use this to force-read a segment for forensic extraction, never as a production setting.

6. Best Practices & The Better Design

The whole class of problem is "a single copy of the data got damaged and there was nothing to fall back to." The better design removes the single point of failure and closes the in-flight window.

# Topic: multiple replicas so a corrupt copy is recoverable
replication.factor=3
min.insync.replicas=2

# Producer: acks=all + idempotence — durable, in-order, no in-flight duplication
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5

Why each lever matters:

• RF ≥ 3, min.insync.replicas=2. Disk bit-rot on one broker is now a re-replication event, not a data-loss event. This is the single highest-value change.
• acks=all + enable.idempotence=true. Idempotence doesn't stop a disk from rotting, but it eliminates the in-flight corruption/duplication window and guarantees the broker's stored CRC reflects exactly one clean write.
• Checksums all the way down. Kafka's CRC catches corruption after the fact; a filesystem with block checksums (ZFS/Btrfs) and ECC RAM catch it at the source. In banking-grade clusters, non-ECC memory is not an option — a single stuck bit in the page cache silently rewrites records the CRC will later reject.
• Honest flush semantics. Kafka relies on the OS page cache and fsync on segment roll. On a hard power loss, un-fsynced pages are lost and you get torn segments. Use disks/arrays with power-loss protection, and don't run brokers on storage that lies about durability.

The "right way" consumer keeps skip-handling as a safety net but treats a corrupt record as an alertable incident routed to a DLQ, not a silent seek(pos+1):

try {
    // normal poll/process/commit
} catch (KafkaException e) {
    if (e.getCause() instanceof CorruptRecordException) {
        metrics.increment("kafka.corrupt_record");
        alerting.page("Corrupt record on " + assignment + " — data loss window opened");
        // record the skipped offset range for downstream reconciliation
        skipCorruptAndContinue(consumer);
    } else {
        throw e;
    }
}

7. How to Prevent It Long-Term

• Monitor CRC failures directly. Alert on broker InvalidMessageCrcRecordsPerSec (or equivalent JMX) and on any consumer-side CorruptRecordException. One occurrence is signal — corruption is never "expected background noise."
• Watch replication health. UnderReplicatedPartitions and OfflineLogDirectoryCount climbing after a corruption event tell you whether re-replication is healing the partition or whether you're down to a single copy.
• Scrub proactively. Periodically run kafka-dump-log.sh --deep-iteration (off-peak, on a follower) against critical topics so you find corruption before a consumer does, and while a clean replica still exists.
• SMART + filesystem checks. Disk health monitoring and ZFS/Btrfs scrubs catch bit-rot at the storage layer weeks before Kafka's CRC does.
• Pin durability config as code. RF, min.insync.replicas, acks, and idempotence belong in your topics-as-code (Strimzi/Terraform) and client config — not left to per-app defaults. Ban check.crcs=false in config lint.
• Chaos-test the recovery path. In a Testcontainers or staging cluster, dd-corrupt a segment and assert that (a) your consumer detects it, alerts, and skips rather than crash-loops, and (b) an RF≥2 replica-delete-and-refetch actually heals the partition. Recovery you haven't rehearsed is recovery you don't have.

8. Key Takeaways

• CorruptRecordException (CORRUPT_MESSAGE, code 2) means stored CRC ≠ computed CRC — the bytes are damaged, not your payload or schema. It's not a poison pill.
• It extends RetriableException, so persistent on-disk corruption makes the fetcher retry the same offset forever. The fix is to seek past it, exactly as the error message says.
• Immediate unblock: reset the group with kafka-consumer-groups.sh --reset-offsets --to-offset <bad+1> or seek(pos+1) in code — but that's data loss, so alert and reconcile.
• Real fix: RF ≥ 3 + min.insync.replicas=2 so a corrupt replica heals by re-fetching a clean copy; RF=1 leaves you no recovery.
• Never set check.crcs=false to make it "go away," and pair Kafka's CRC with ECC RAM + checksumming storage to stop corruption at the source.