Kafka SerializationException: Fixing Poison Pill Records

1. The Error

Your consumer dies, restarts, dies again on the exact same offset. Forever. This is the log:

org.apache.kafka.common.errors.RecordDeserializationException: Error deserializing VALUE for partition orders-2 at offset 4173. If needed, please seek past the record to continue consumption.
	at org.apache.kafka.clients.consumer.internals.CompletedFetch.newRecordDeserializationException(CompletedFetch.java:346)
	at org.apache.kafka.clients.consumer.internals.CompletedFetch.parseRecord(CompletedFetch.java:330)
	at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:284)
	at org.apache.kafka.clients.consumer.internals.FetchCollector.fetchRecords(FetchCollector.java:168)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:874)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1602)
Caused by: org.apache.kafka.common.errors.SerializationException: Can't deserialize data [[110, 111, 116, 45, 106, 115, 111, 110]] from topic [orders]
	at org.springframework.kafka.support.serializer.JsonDeserializer.deserialize(JsonDeserializer.java:588)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'not': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (byte[])"not-json"; line: 1, column: 4]

On kafka-clients before 2.8 the top-level exception is a plain SerializationException with no partition/offset in the type (KIP-334 added RecordDeserializationException). Same disease, worse diagnostics.

Typical setup: kafka-clients 3.9/4.x, Spring Boot 3.5.x with spring-kafka 3.3.x, JsonDeserializer (or an Avro deserializer pointed at the wrong schema), running against Docker locally or a managed cluster in early prod. The trigger is one record on the topic that your deserializer cannot parse — a poison pill.

The nasty part: this exception is thrown from poll(), not from your listener. Spring's retry/error handling for listener exceptions never sees it. The container restarts, fetches from the last committed offset, hits the same record, throws again. That partition is now dead — lag climbs while the other partitions drain fine.

2. How to Reproduce It

Single-node KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:4.0.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: INTERNAL://:19092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:19092 --create --topic orders --partitions 3

Consumer — Spring Boot 3.5.x, spring-kafka 3.3.x:

public record Order(String orderId, java.math.BigDecimal amount) {}

@Component
public class OrderListener {

    @KafkaListener(topics = "orders", groupId = "order-service")
    public void onOrder(Order order) {
        System.out.println("Processing " + order.orderId());
    }
}

# application.yml — the vulnerable config
spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.value.default.type: com.example.orders.Order
        spring.json.trusted.packages: com.example.orders

Send one valid record, then the poison pill:

docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:19092 --topic orders
>{"orderId":"o-1","amount":42.50}
>not-json

The first record processes. The second throws RecordDeserializationException, the container logs the stack trace, restarts, and throws again — every few seconds, indefinitely. Confirm the stuck partition:

docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:19092 --describe --group order-service

Lag on one partition never decreases. That's the signature.

Environment-specific triggers you'll see in the wild: someone pastes raw JSON with a trailing comma into kafka-console-producer against a prod topic; a producer team switches from JSON to Avro mid-topic without a migration plan; a different service starts writing to the same topic with an incompatible payload; a byte-level corruption from a misconfigured MirrorMaker transform.

3. Why It Happens — Surface Level

Deserialization runs inside the consumer client, during poll(), before the record ever reaches your code. When the configured Deserializer throws, the client wraps it in RecordDeserializationException and propagates it out of poll(). The consumer's position for that partition does not advance past the bad record, and nothing was committed — so the next fetch returns the same record.

Your listener-level retries, @RetryableTopic, and DefaultErrorHandler all operate on exceptions thrown by the listener. This exception happens a layer below them. Unless something either skips the record or makes deserialization stop throwing, the partition is permanently blocked.

4. Why It Happens — Under the Hood

The fetch path: the broker returns raw record batches; the client's FetchCollector/CompletedFetch iterates them and calls Deserializer.deserialize() per record while assembling the ConsumerRecords for poll(). Kafka made a deliberate design call here — the client does not silently skip undeserializable data, because in a financial or audit context "skip silently" is data loss. It throws and leaves the decision to you.

Since KIP-334 (kafka-clients 2.8+), the exception carries topicPartition() and offset(), precisely so you can implement seek-past logic. The consumer's in-memory position is left at the failed offset. Auto-commit doesn't save you: commits only cover offsets already returned by poll(), and this record never made it out of poll().

Spring Kafka's KafkaMessageListenerContainer catches the exception, logs it, and by default stops/restarts the consumer thread. Restart → rejoin group → fetch committed offset → same record → same exception. If you run multiple instances, the rebalance from each crash-restart cycle also stalls the healthy members — with an eager assignor this cascades into churn across the whole group.

One more subtlety: a poison pill in the key is worse than one in the value. ErrorHandlingDeserializer handles both, but if you only wrap the value deserializer (the common copy-paste), a corrupt key still kills you.

5. The Fix

Fix 1 (Spring Kafka — the right one): ErrorHandlingDeserializer

Wrap your real deserializer so failures return a null payload with the exception in a header, instead of throwing inside poll():

 spring:
   kafka:
     consumer:
       key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
-      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
+      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
       properties:
+        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
         spring.json.value.default.type: com.example.orders.Order
         spring.json.trusted.packages: com.example.orders

Now the record reaches the container, Spring detects the failed deserialization, and raises it as a listener-level error — which your error handler can process. Route failures to a dead letter topic:

@Configuration
public class KafkaErrorConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        var recoverer = new DeadLetterPublishingRecoverer(template);
        // Deserialization failures are not transient — don't retry them.
        var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2));
        handler.addNotRetryableExceptions(DeserializationException.class);
        return handler;
    }
}

DeadLetterPublishingRecoverer publishes the raw bytes to orders.DLT (same partition by default — create the DLT with at least as many partitions as the source, or pass a custom destination resolver). The consumer commits and moves on. Partition unblocked, evidence preserved.

If you wrap the key too:

spring.kafka.consumer.key-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.properties.spring.deserializer.key.delegate.class=org.apache.kafka.common.serialization.StringDeserializer

Fix 2 (plain kafka-clients): catch and seek past

No Spring? Use the KIP-334 metadata:

while (running) {
    ConsumerRecords<String, Order> records;
    try {
        records = consumer.poll(Duration.ofMillis(500));
    } catch (RecordDeserializationException e) {
        log.error("Poison pill at {} offset {} — skipping", e.topicPartition(), e.offset(), e);
        // Optionally: fetch the raw bytes with a byte-array consumer and publish to a DLT here.
        consumer.seek(e.topicPartition(), e.offset() + 1);
        continue;
    }
    process(records);
}

Blunt but effective. The record is dropped unless you capture it yourself — acceptable for a POC, not for anything with audit requirements.

Fix 3 (one-off unblock in prod, right now)

If the fleet is down and you just need the partition moving while the code fix ships:

# Stop all consumers in the group first — offsets can only be reset on an empty group
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group order-service --topic orders:2 \
  --reset-offsets --shift-by 1 --execute

This skips exactly one record on partition 2. Log which offset you skipped; someone will ask.

When to use which: Fix 1 for any Spring service — it should be your default consumer config, not an incident response. Fix 2 for raw-client services. Fix 3 is a tourniquet, never the fix.

6. Best Practices & The Better Design

The class of problem is "consumer trusts the wire format." The better design assumes it can't.

ErrorHandlingDeserializer + DLT as the paved road. Every Spring Kafka consumer in the org gets this config by default. Not per-team discretion — a shared starter or config library:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.value.default.type: com.example.orders.Order
        spring.json.trusted.packages: com.example.orders

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var handler = new DefaultErrorHandler(
        new DeadLetterPublishingRecoverer(template,
            (rec, ex) -> new TopicPartition(rec.topic() + ".DLT", rec.partition())),
        new FixedBackOff(1000L, 2));
    handler.addNotRetryableExceptions(DeserializationException.class,
        MessageConversionException.class);
    return handler;
}

Stop garbage at produce time with a schema registry. JSON on a bare topic means every producer is one typo away from poisoning every consumer. Avro/Protobuf with Confluent Schema Registry (or Karapace/AWS Glue) moves validation to the producer: an incompatible payload fails at send(), in the producing team's service, where it belongs. Compatibility mode BACKWARD on the subject means old consumers keep working through schema evolution. This eliminates the "producer changed the format" poison pill entirely — you're left only with genuine corruption, which the DLT absorbs.

For maximum control: consume bytes, deserialize yourself. High-stakes pipelines (payments, ledger events) sometimes use ByteArrayDeserializer and do explicit deserialization inside the listener, in a try/catch, with full context. It's more code, but the failure handling is ordinary application code — no framework layering to reason about during an incident.

7. How to Prevent It Long-Term

Alert on the signature, not just total lag. Total group lag hides a single stuck partition behind healthy ones. Alert on max lag per partition (kafka_consumergroup_lag by partition in Burrow/kafka-lag-exporter/Prometheus) with a "lag monotonically increasing for N minutes on one partition" rule. That pattern is almost always a poison pill or a hot key.

Monitor DLT depth and age. An empty DLT config you never look at is silent data loss with extra steps. Alert when orders.DLT receives messages; have a documented replay path (a small replayer service or kafka-console-consumer + fixed producer) and an owner.

Contract-test the wire format in CI. Producer and consumer both test against the registered schema (Pact, or plain schema-registry compatibility checks in the pipeline: mvn schema-registry:test-compatibility). A breaking change fails the producer's build, not the consumer's on-call.

Lock down prod topics. Most real-world poison pills I've debugged came from a human with kafka-console-producer and good intentions. ACLs: application principals only for WRITE on prod topics. Humans get a sandbox cluster.

Chaos-test it. Once a quarter, inject a garbage record into a staging topic and verify: consumer keeps moving, DLT receives the record with exception headers, the alert fires. If any of those three doesn't happen, you have the same outage waiting in prod.

8. Key Takeaways

• RecordDeserializationException is thrown from poll(), before your listener — listener-level retries and @RetryableTopic can't catch it, and the consumer will loop on the same offset forever.
• The default Spring Kafka JsonDeserializer config is a poison-pill trap. Wrap key and value in ErrorHandlingDeserializer — make it the org-wide default, not incident-driven config.
• Pair it with DefaultErrorHandler + DeadLetterPublishingRecoverer, and mark DeserializationException as not-retryable — a malformed record never deserializes on retry #3.
• On raw kafka-clients, catch RecordDeserializationException and seek(partition, offset + 1) (KIP-334, clients ≥ 2.8).
• Alert on per-partition lag that only goes up, and on DLT depth. A schema registry with producer-side validation kills this class of bug at the source.