Kafka SerializationException: Can't Convert Value of Class Fix

1. The Error

Your producer dies on the first send():

Exception in thread "main" org.apache.kafka.common.errors.SerializationException: Can't convert value of class com.example.orders.OrderEvent to class org.apache.kafka.common.serialization.StringSerializer specified in value.serializer
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1090)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:1017)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:962)
	at com.example.orders.OrderProducer.main(OrderProducer.java:24)
Caused by: java.lang.ClassCastException: class com.example.orders.OrderEvent cannot be cast to class java.lang.String (com.example.orders.OrderEvent is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
	at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:29)
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1087)
	... 3 more

The key-side twin is identical except for the first line: Can't convert key of class ... specified in key.serializer.

Where you'll hit it:

• kafka-clients: any version — the message text has been stable for years (verified against 3.x/4.x)
• Spring Boot / Spring Kafka: Boot 3.x with spring-kafka 3.x, where KafkaTemplate.send() throws the same exception because Boot's autoconfigured default serializer is StringSerializer
• Setup: almost always a POC or a brand-new service — this error means the producer has never successfully sent this record type, ever. It is a wiring bug, not an infrastructure incident.

Note what it is not: it is not the consumer-side poison-pill problem (RecordDeserializationException inside poll()). This one fires in the producer, synchronously, before a single byte reaches the broker.

2. How to Reproduce It (step-by-step)

Single-broker KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: INTERNAL://kafka:29092,CONTROLLER://kafka:29093,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d

Producer with the classic mistake — a POJO value, but the serializer config copy-pasted from a hello-world string example:

// build.gradle: implementation 'org.apache.kafka:kafka-clients:3.9.1'
package com.example.orders;

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class OrderProducer {

    record OrderEvent(String orderId, long amountCents) {}

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // Compiles without a single warning:
        KafkaProducer<String, Object> producer = new KafkaProducer<>(props);

        producer.send(new ProducerRecord<>("orders",
                "o-42", new OrderEvent("o-42", 19_99)));   // 💥 throws here, synchronously
        producer.close();
    }
}

Run it. The send() call itself throws — no callback fires, no retry happens, nothing hits the broker. kafka-console-consumer.sh --topic orders --from-beginning shows an empty topic.

Spring Boot reproduction is even easier because it's the default:

@Autowired KafkaTemplate<String, Object> kafkaTemplate;   // Boot default: StringSerializer for both
kafkaTemplate.send("orders", new OrderEvent("o-42", 1999)); // same SerializationException

No environment dependency: no Docker networking, no broker config, no version matrix. The record type and the serializer class disagree; that's the whole trigger.

3. Why It Happens — Surface Level

value.serializer tells the producer which class turns your value object into bytes. You configured StringSerializer, whose serialize method accepts a String. You handed it an OrderEvent. Java can't cast OrderEvent to String, throws ClassCastException, and KafkaProducer.doSend() wraps it into the SerializationException you're staring at — including, helpfully, both class names so you can see the exact mismatch.

The fix is to make the two agree: either configure a serializer that understands your type (JSON/Avro/Protobuf), or convert the value to a String yourself before send(). Nothing about brokers, topics, or networking is involved.

4. Why It Happens — Under the Hood

The interesting question is why the compiler let you do this. Three mechanics stack up:

1. Config-string wiring erases the type relationship. When serializers are passed as class names in the config map, KafkaProducer instantiates them reflectively (Utils.newInstance() via config.getConfiguredInstance()). At that point the producer holds a Serializer<K> and Serializer<V> whose actual type parameters are unknown to the compiler — an unchecked assignment happens inside the client. KafkaProducer<String, Object> with a StringSerializer value serializer is nonsense, but it's runtime-only nonsense.

2. Generics are erased; the cast happens inside the serializer. StringSerializer implements Serializer<String>. After erasure, the bridge method accepts Object and casts to String before calling the real serialize(String topic, String data). Your OrderEvent sails through every compile-time check and detonates on that cast — the first line of the serializer. This is why the stack trace bottoms out at StringSerializer.serialize.

3. The exception is synchronous and pre-accumulator. In doSend(), key and value serialization happen before partitioning and before the record is appended to the RecordAccumulator. The catch block is explicit:

} catch (ClassCastException cce) {
    throw new SerializationException("Can't convert value of class " + record.value().getClass().getName()
        + " to class " + valueSerializer.getClass().getName()
        + " specified in value.serializer", cce);
}

Consequences worth internalizing: the producer's retry machinery (retries, delivery.timeout.ms, idempotence) never engages — those govern the network path, and this record never got that far. The Callback is never invoked either; the exception surfaces on the calling thread. If your code only handles errors in the callback (the common pattern from the batch-expiry world), a fire-and-forget wrapper like void publish(Object e) { producer.send(rec(e)); } will still crash — but a wrapper that swallows exceptions around send() will silently drop every single record. In Spring, KafkaTemplate.send() throws before returning its CompletableFuture, so .whenComplete() handlers never see it.

Why Spring Boot makes this the default trap: Boot's KafkaAutoConfiguration defaults both spring.kafka.producer.key-serializer and value-serializer to StringSerializer. Every tutorial that sends strings works out of the box; the first time you send a domain object, you get this exception. The consumer-side mirror (JsonDeserializer + trusted packages) fails differently — as a poison pill in poll() — which is a separate article in this series.

5. The Fix

Fix A — configure a JSON serializer (POC / internal topics)

Raw client, using spring-kafka's JsonSerializer (or roll your own Jackson wrapper — it's ~15 lines):

 props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
-props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
+props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.springframework.kafka.support.serializer.JsonSerializer");

Spring Boot application.yml:

 spring:
   kafka:
     bootstrap-servers: localhost:9092
     producer:
       key-serializer: org.apache.kafka.common.serialization.StringSerializer
+      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

Minimal hand-rolled serializer if you don't want the spring-kafka dependency in a plain Java service:

public class JacksonSerializer<T> implements Serializer<T> {
    private static final ObjectMapper MAPPER = new ObjectMapper();
    @Override public byte[] serialize(String topic, T data) {
        if (data == null) return null;
        try {
            return MAPPER.writeValueAsBytes(data);
        } catch (JsonProcessingException e) {
            throw new SerializationException("JSON serialization failed for topic " + topic, e);
        }
    }
}

Fix B — serialize explicitly at the call site (quick unblock)

-producer.send(new ProducerRecord<>("orders", "o-42", event));
+producer.send(new ProducerRecord<>("orders", "o-42", mapper.writeValueAsString(event)));

Fine for a spike; in a real service you want the serializer owned by producer config, not scattered across call sites.

Fix C — constructor-injected serializers (do this everywhere)

This is the fix for the class of bug, not just this instance. KafkaProducer has a constructor that takes serializer instances, which restores compile-time type checking:

-props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
-props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
-KafkaProducer<String, Object> producer = new KafkaProducer<>(props);
+KafkaProducer<String, OrderEvent> producer = new KafkaProducer<>(
+        props,
+        new StringSerializer(),
+        new JacksonSerializer<OrderEvent>());

Now producer.send(new ProducerRecord<>("orders", "o-42", "some string")) is a compile error. The config-string style can never give you that.

When to use which: Fix B to unblock a spike today; Fix A when the topic is internal to one team and JSON is acceptable; Fix C as a standing convention regardless of format. For anything crossing team boundaries or living longer than a quarter, go to schema-backed serialization (next section) — JSON-without-schema is how you end up writing the Schema Registry 409 article's incident report later.

6. Best Practices & The Better Design

Precise generics, one producer per value type. KafkaProducer<String, Object> (and KafkaTemplate<String, Object>) is the root enabler of this bug — it tells the compiler "don't check anything." Declare KafkaTemplate<String, OrderEvent> per event family. In Spring, build them explicitly instead of leaning on the autoconfigured template:

@Configuration
public class KafkaProducerConfig {

    @Bean
    public ProducerFactory<String, OrderEvent> orderProducerFactory(KafkaProperties kafkaProperties) {
        Map<String, Object> props = kafkaProperties.buildProducerProperties(null);
        // Serializer *instances* — type-checked, and JsonSerializer gets your ObjectMapper:
        return new DefaultKafkaProducerFactory<>(
                props,
                new StringSerializer(),
                new JsonSerializer<>(new ObjectMapper().registerModule(new JavaTimeModule())));
    }

    @Bean
    public KafkaTemplate<String, OrderEvent> orderKafkaTemplate(
            ProducerFactory<String, OrderEvent> orderProducerFactory) {
        return new KafkaTemplate<>(orderProducerFactory);
    }
}

Injecting KafkaTemplate<String, OrderEvent> now fails at compile time if someone sends the wrong type, and fails at context startup if the bean wiring is wrong — both strictly better than failing on the first production send().

Schema-backed serialization for anything durable. JSON with no contract just moves the type mismatch from your producer to every consumer. The production-grade design is Avro or Protobuf with a schema registry:

value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url=http://schema-registry:8081
auto.register.schemas=false
use.latest.version=true

With generated SpecificRecord classes, the type agreement between your code and the wire format is enforced by codegen, and evolution is governed by compatibility rules instead of hope.

One convention to ban the bug: serializers are always passed as instances (Fix C style), never as config strings, except where an ecosystem forces strings (Connect, Streams default.value.serde). Put it in the team's Kafka config module and stop thinking about it.

7. How to Prevent It Long-Term

Testcontainers smoke test that actually sends your real event type. Unit tests with MockProducer won't catch this — MockProducer happily accepts any object unless you construct it with real serializers (so do that). The reliable gate is one integration test per producer:

@Test
void publishesOrderEvent() throws Exception {
    kafkaTemplate.send("orders", "o-1", new OrderEvent("o-1", 500)).get(10, TimeUnit.SECONDS);
    // consume back with the matching deserializer and assert round-trip equality
}

This single test catches serializer mismatch, missing trusted packages on the consumer side, and schema incompatibilities — the entire wiring class — before any deploy.

Static checks. Forbid KafkaProducer<Object, Object>, KafkaTemplate<String, Object>, and raw KafkaTemplate via ArchUnit or Error Prone in CI. Grep-level enforcement is enough: the offending declarations are syntactically obvious.

Fail at startup, not at first send. A tiny SmartLifecycle bean that serializes a canary instance of each event type through the configured serializer at boot turns "first order of the day fails" into "deploy fails health check" — vastly cheaper.

Log-based detection. This exception is synchronous, so it shows up in your service logs, not broker logs and not producer metrics (record-error-rate never increments — the record was never appended). Alert on SerializationException in application logs; don't expect Kafka-side monitoring to see it.

8. Key Takeaways

• Can't convert value of class X to class Y specified in value.serializer = your record type and configured serializer disagree. It's a wiring bug; the broker is never involved.
• The exception is synchronous — thrown from send() itself. Callbacks never fire, retries never engage, record-error-rate stays at zero. Handle it on the calling thread.
• Spring Boot defaults both producer serializers to StringSerializer; the first domain object you send reproduces this exactly. Set spring.kafka.producer.value-serializer explicitly.
• Pass serializer instances to the KafkaProducer/DefaultKafkaProducerFactory constructor instead of class-name strings — that turns this whole bug class into a compile error.
• One Testcontainers round-trip test per producer with the real event type catches serializer mismatch, deserializer mismatch, and schema drift before production ever sees them.