> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka SerializationException: Fixing Poison Pill Records - URL: https://www.ggorantala.dev/kafka-serializationexception-poison-pill/ - Published: 2026-02-01T08:17:00.000Z - Updated: 2026-08-29T08:18:19.000Z - Description: Kafka SerializationException: "Error deserializing key/value" crashes your consumer in a loop. Fix it with ErrorHandlingDeserializer and a dead letter topic. - Author: Gopi Gorantala - Tags: kafka-errors, serializationexception, recorddeserializationexception, spring-kafka, afka-consumer, dead-letter-topic, Java ## 1\. The Error Your consumer dies, restarts, dies again on the exact same offset. Forever. This is the log: ```text org.apache.kafka.common.errors.RecordDeserializationException: Error deserializing VALUE for partition orders-2 at offset 4173. If needed, please seek past the record to continue consumption. at org.apache.kafka.clients.consumer.internals.CompletedFetch.newRecordDeserializationException(CompletedFetch.java:346) at org.apache.kafka.clients.consumer.internals.CompletedFetch.parseRecord(CompletedFetch.java:330) at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:284) at org.apache.kafka.clients.consumer.internals.FetchCollector.fetchRecords(FetchCollector.java:168) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:874) at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1602) Caused by: org.apache.kafka.common.errors.SerializationException: Can't deserialize data [[110, 111, 116, 45, 106, 115, 111, 110]] from topic [orders] at org.springframework.kafka.support.serializer.JsonDeserializer.deserialize(JsonDeserializer.java:588) Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'not': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (byte[])"not-json"; line: 1, column: 4] ``` On kafka-clients before 2.8 the top-level exception is a plain `SerializationException` with no partition/offset in the type (KIP-334 added `RecordDeserializationException`). Same disease, worse diagnostics. Typical setup: kafka-clients 3.9/4.x, Spring Boot 3.5.x with spring-kafka 3.3.x, `JsonDeserializer` (or an Avro deserializer pointed at the wrong schema), running against Docker locally or a managed cluster in early prod. The trigger is one record on the topic that your deserializer cannot parse — a **poison pill**. The nasty part: this exception is thrown from `poll()`, not from your listener. Spring's retry/error handling for listener exceptions never sees it. The container restarts, fetches from the last committed offset, hits the same record, throws again. That partition is now dead — lag climbs while the other partitions drain fine. ## 2\. How to Reproduce It Single-node KRaft cluster: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:4.0.0 ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_LISTENERS: INTERNAL://:19092,CONTROLLER://:9093,EXTERNAL://:9092 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ``` ```bash docker compose up -d docker exec kafka /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:19092 --create --topic orders --partitions 3 ``` Consumer — Spring Boot 3.5.x, spring-kafka 3.3.x: ```java public record Order(String orderId, java.math.BigDecimal amount) {} @Component public class OrderListener { @KafkaListener(topics = "orders", groupId = "order-service") public void onOrder(Order order) { System.out.println("Processing " + order.orderId()); } } ``` ```yaml # application.yml — the vulnerable config spring: kafka: bootstrap-servers: localhost:9092 consumer: key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: spring.json.value.default.type: com.example.orders.Order spring.json.trusted.packages: com.example.orders ``` Send one valid record, then the poison pill: ```bash docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:19092 --topic orders >{"orderId":"o-1","amount":42.50} >not-json ``` The first record processes. The second throws `RecordDeserializationException`, the container logs the stack trace, restarts, and throws again — every few seconds, indefinitely. Confirm the stuck partition: ```bash docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:19092 --describe --group order-service ``` Lag on one partition never decreases. That's the signature. Environment-specific triggers you'll see in the wild: someone pastes raw JSON with a trailing comma into `kafka-console-producer` against a prod topic; a producer team switches from JSON to Avro mid-topic without a migration plan; a different service starts writing to the same topic with an incompatible payload; a byte-level corruption from a misconfigured MirrorMaker transform. ## 3\. Why It Happens — Surface Level Deserialization runs inside the consumer client, during `poll()`, before the record ever reaches your code. When the configured `Deserializer` throws, the client wraps it in `RecordDeserializationException` and propagates it out of `poll()`. The consumer's position for that partition does **not** advance past the bad record, and nothing was committed — so the next fetch returns the same record. Your listener-level retries, `@RetryableTopic`, and `DefaultErrorHandler` all operate on exceptions thrown *by the listener*. This exception happens a layer below them. Unless something either skips the record or makes deserialization stop throwing, the partition is permanently blocked. ## 4\. Why It Happens — Under the Hood The fetch path: the broker returns raw record batches; the client's `FetchCollector`/`CompletedFetch` iterates them and calls `Deserializer.deserialize()` per record while assembling the `ConsumerRecords` for `poll()`. Kafka made a deliberate design call here — the client does not silently skip undeserializable data, because in a financial or audit context "skip silently" is data loss. It throws and leaves the decision to you. Since KIP-334 (kafka-clients 2.8+), the exception carries `topicPartition()` and `offset()`, precisely so you *can* implement seek-past logic. The consumer's in-memory position is left at the failed offset. Auto-commit doesn't save you: commits only cover offsets already returned by `poll()`, and this record never made it out of `poll()`. Spring Kafka's `KafkaMessageListenerContainer` catches the exception, logs it, and by default stops/restarts the consumer thread. Restart → rejoin group → fetch committed offset → same record → same exception. If you run multiple instances, the rebalance from each crash-restart cycle also stalls the *healthy* members — with an eager assignor this cascades into churn across the whole group. One more subtlety: a poison pill in the **key** is worse than one in the value. `ErrorHandlingDeserializer` handles both, but if you only wrap the value deserializer (the common copy-paste), a corrupt key still kills you. ## 5\. The Fix ### Fix 1 (Spring Kafka — the right one): `ErrorHandlingDeserializer` Wrap your real deserializer so failures return a `null` payload with the exception in a header, instead of throwing inside `poll()`: ```diff spring: kafka: consumer: key-deserializer: org.apache.kafka.common.serialization.StringDeserializer - value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer + value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: + spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer spring.json.value.default.type: com.example.orders.Order spring.json.trusted.packages: com.example.orders ``` Now the record reaches the container, Spring detects the failed deserialization, and raises it as a listener-level error — which your error handler *can* process. Route failures to a dead letter topic: ```java @Configuration public class KafkaErrorConfig { @Bean public DefaultErrorHandler errorHandler(KafkaTemplate template) { var recoverer = new DeadLetterPublishingRecoverer(template); // Deserialization failures are not transient — don't retry them. var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2)); handler.addNotRetryableExceptions(DeserializationException.class); return handler; } } ``` `DeadLetterPublishingRecoverer` publishes the raw bytes to `orders.DLT` (same partition by default — create the DLT with at least as many partitions as the source, or pass a custom destination resolver). The consumer commits and moves on. Partition unblocked, evidence preserved. If you wrap the key too: ```properties spring.kafka.consumer.key-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer spring.kafka.consumer.properties.spring.deserializer.key.delegate.class=org.apache.kafka.common.serialization.StringDeserializer ``` ### Fix 2 (plain kafka-clients): catch and seek past No Spring? Use the KIP-334 metadata: ```java while (running) { ConsumerRecords records; try { records = consumer.poll(Duration.ofMillis(500)); } catch (RecordDeserializationException e) { log.error("Poison pill at {} offset {} — skipping", e.topicPartition(), e.offset(), e); // Optionally: fetch the raw bytes with a byte-array consumer and publish to a DLT here. consumer.seek(e.topicPartition(), e.offset() + 1); continue; } process(records); } ``` Blunt but effective. The record is dropped unless you capture it yourself — acceptable for a POC, not for anything with audit requirements. ### Fix 3 (one-off unblock in prod, right now) If the fleet is down and you just need the partition moving while the code fix ships: ```bash # Stop all consumers in the group first — offsets can only be reset on an empty group kafka-consumer-groups.sh --bootstrap-server broker:9092 \ --group order-service --topic orders:2 \ --reset-offsets --shift-by 1 --execute ``` This skips exactly one record on partition 2\. Log which offset you skipped; someone will ask. **When to use which:** Fix 1 for any Spring service — it should be your default consumer config, not an incident response. Fix 2 for raw-client services. Fix 3 is a tourniquet, never the fix. ## 6\. Best Practices & The Better Design The class of problem is "consumer trusts the wire format." The better design assumes it can't. **ErrorHandlingDeserializer + DLT as the paved road.** Every Spring Kafka consumer in the org gets this config by default. Not per-team discretion — a shared starter or config library: ```yaml spring: kafka: bootstrap-servers: localhost:9092 consumer: key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer spring.json.value.default.type: com.example.orders.Order spring.json.trusted.packages: com.example.orders ``` ```java @Bean public DefaultErrorHandler errorHandler(KafkaTemplate template) { var handler = new DefaultErrorHandler( new DeadLetterPublishingRecoverer(template, (rec, ex) -> new TopicPartition(rec.topic() + ".DLT", rec.partition())), new FixedBackOff(1000L, 2)); handler.addNotRetryableExceptions(DeserializationException.class, MessageConversionException.class); return handler; } ``` **Stop garbage at produce time with a schema registry.** JSON on a bare topic means every producer is one typo away from poisoning every consumer. Avro/Protobuf with Confluent Schema Registry (or Karapace/AWS Glue) moves validation to the producer: an incompatible payload fails at `send()`, in the producing team's service, where it belongs. Compatibility mode `BACKWARD` on the subject means old consumers keep working through schema evolution. This eliminates the "producer changed the format" poison pill entirely — you're left only with genuine corruption, which the DLT absorbs. **For maximum control: consume bytes, deserialize yourself.** High-stakes pipelines (payments, ledger events) sometimes use `ByteArrayDeserializer` and do explicit deserialization inside the listener, in a try/catch, with full context. It's more code, but the failure handling is ordinary application code — no framework layering to reason about during an incident. ## 7\. How to Prevent It Long-Term **Alert on the signature, not just total lag.** Total group lag hides a single stuck partition behind healthy ones. Alert on max lag *per partition* (`kafka_consumergroup_lag` by partition in Burrow/kafka-lag-exporter/Prometheus) with a "lag monotonically increasing for N minutes on one partition" rule. That pattern is almost always a poison pill or a hot key. **Monitor DLT depth and age.** An empty DLT config you never look at is silent data loss with extra steps. Alert when `orders.DLT` receives messages; have a documented replay path (a small replayer service or `kafka-console-consumer` \+ fixed producer) and an owner. **Contract-test the wire format in CI.** Producer and consumer both test against the registered schema (Pact, or plain schema-registry compatibility checks in the pipeline: `mvn schema-registry:test-compatibility`). A breaking change fails the producer's build, not the consumer's on-call. **Lock down prod topics.** Most real-world poison pills I've debugged came from a human with `kafka-console-producer` and good intentions. ACLs: application principals only for `WRITE` on prod topics. Humans get a sandbox cluster. **Chaos-test it.** Once a quarter, inject a garbage record into a staging topic and verify: consumer keeps moving, DLT receives the record with exception headers, the alert fires. If any of those three doesn't happen, you have the same outage waiting in prod. ## 8\. Key Takeaways - `RecordDeserializationException` is thrown from `poll()`, **before** your listener — listener-level retries and `@RetryableTopic` can't catch it, and the consumer will loop on the same offset forever. - The default Spring Kafka `JsonDeserializer` config is a poison-pill trap. Wrap key **and** value in `ErrorHandlingDeserializer` — make it the org-wide default, not incident-driven config. - Pair it with `DefaultErrorHandler` \+ `DeadLetterPublishingRecoverer`, and mark `DeserializationException` as not-retryable — a malformed record never deserializes on retry #3. - On raw kafka-clients, catch `RecordDeserializationException` and `seek(partition, offset + 1)` (KIP-334, clients ≥ 2.8). - Alert on per-partition lag that only goes up, and on DLT depth. A schema registry with producer-side validation kills this class of bug at the source.