Kafka SerializationException: Schema Not Found (40403) Fix

Your consumer was fine yesterday. Today, in a different environment, every record you poll blows up with a SerializationException complaining about a schema ID that plainly exists — you can see it in your other cluster. The consumer is in a crash loop, offsets aren't advancing, and lag is climbing.

This is not a compatibility problem and not a poison-pill payload. The 5-byte Confluent wire header on each record carries a schema ID, the deserializer takes that ID to a Schema Registry, and the registry says it has never heard of it. Almost always, the consumer is talking to a different registry than the one the producer registered against.

1. The Error

The canonical stack trace on the consumer, wrapped up through poll():

org.apache.kafka.common.errors.SerializationException: Error retrieving Avro schema for id 42
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
    Schema not found; error code: 40403
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:308)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:395)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:788)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaByIdFromRegistry(CachedSchemaRegistryClient.java:302)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaBySubjectAndId(CachedSchemaRegistryClient.java:412)
    at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:...)

You will see one of a small family of REST error codes, and the distinction matters:

• Schema not found; error code: 40403 — the deserializer asked GET /schemas/ids/{id} and no schema with that global ID exists in this registry.
• Subject not found; error code: 40401 — a lookup keyed by subject (GET /subjects/{subject}/versions/{version}) found no such subject. Common with use.latest.version=true, ID-less lookups, or a subject-naming-strategy mismatch.
• Version not found; error code: 40402 — the subject exists but the requested version doesn't (usually a hard-deleted version).

In Spring Kafka the same failure arrives wrapped one more time — ListenerExecutionFailedException on top, or, because deserialization happens inside poll() before your listener runs, as a container-level error that redelivers the same offset forever.

Where it shows up: Kafka 3.9/4.0 brokers, kafka-avro-serializer / kafka-schema-registry-client in the 7.x–8.2.x line, Confluent Platform 7.x–8.x or Confluent Cloud. The mechanics are identical for the Protobuf and JSON Schema deserializers — only the exception message noun changes.

2. How to Reproduce It

Stand up a broker, a registry, produce one record so a schema gets registered, then point a consumer at a second, empty registry — exactly what happens when dev and staging each run their own.

# docker-compose.yml — Kafka 3.9 KRaft + two Schema Registries
services:
  kafka:
    image: apache/kafka:3.9.0
    ports: ["9092:9092", "29092:29092"]
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://kafka:9092,CONTROLLER://kafka:9093,EXTERNAL://0.0.0.0:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  schema-registry-a:
    image: confluentinc/cp-schema-registry:7.9.0
    depends_on: [kafka]
    ports: ["8081:8081"]
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry-a
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_KAFKASTORE_TOPIC: _schemas_a

  schema-registry-b:
    image: confluentinc/cp-schema-registry:7.9.0
    depends_on: [kafka]
    ports: ["8082:8081"]
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry-b
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_KAFKASTORE_TOPIC: _schemas_b

Produce a record against registry A (port 8081):

Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
p.put("schema.registry.url", "http://localhost:8081"); // registry A

Schema schema = SchemaBuilder.record("OrderCreated").namespace("com.example.orders")
        .fields().requiredString("orderId").requiredLong("amount").endRecord();
GenericRecord rec = new GenericData.Record(schema);
rec.put("orderId", "o-1"); rec.put("amount", 4200L);

try (var producer = new KafkaProducer<String, GenericRecord>(p)) {
    producer.send(new ProducerRecord<>("orders", "o-1", rec)).get();
}

Registry A now holds schema ID 1. Confirm it:

curl -s http://localhost:8081/schemas/ids/1        # 200, returns the schema
curl -s http://localhost:8082/schemas/ids/1        # {"error_code":40403,"message":"Schema not found"}

Now consume, pointing at registry B (port 8082), which is empty:

Properties c = new Properties();
c.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
c.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-consumer");
c.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
c.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
c.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
c.put("schema.registry.url", "http://localhost:8082"); // registry B — WRONG

try (var consumer = new KafkaConsumer<String, GenericRecord>(c)) {
    consumer.subscribe(List.of("orders"));
    while (true) {
        var records = consumer.poll(Duration.ofMillis(500)); // throws here
        records.forEach(r -> System.out.println(r.value()));
    }
}

poll() throws SerializationException: Error retrieving Avro schema for id 1 ... Schema not found; error code: 40403. Same broker, same topic, same bytes — the only variable is which registry the consumer resolves the ID against.

3. Why It Happens — Surface Level

The Confluent wire format does not carry the schema. It carries a reference to it: 1 magic byte (0x0) plus a 4-byte big-endian global schema ID. On deserialize, the client reads that ID and calls its configured Schema Registry — GET /schemas/ids/{id} — to fetch the writer schema. If the registry it is configured with does not contain that ID, you get a 404 → 40403, wrapped as SerializationException.

The ID is only meaningful within the registry that minted it. Global IDs are not portable across registries. Producer registered ID 42 in registry A; your consumer is asking registry B, which assigns IDs from its own sequence and has no entry 42. The bytes on the topic are perfectly valid; the lookup target is wrong.

4. Why It Happens — Under the Hood

AbstractKafkaAvroDeserializer.deserialize() pulls the ID off the buffer and hands it to a CachedSchemaRegistryClient. That client keeps an in-memory idToSchemaCache; on a miss it issues getSchemaByIdFromRegistry(id) → RestService.getId(id) → GET /schemas/ids/{id}. A non-2xx REST response is thrown as RestClientException carrying the body's error_code, and the deserializer rethrows it as SerializationException. Because the ID isn't cached, every record with that ID re-hits the registry and re-throws — there is no negative caching of "not found."

Global IDs come from the registry's own allocator. Under the default _schemas Kafka-backed store, the registry is a compacted-topic-backed KV store; the ID sequence is derived from that log. Two independent registries (separate _schemas topics, or separate clusters) have independent sequences, so the same logical schema can be ID 1 in one and ID 7 in another, and an ID from one is simply absent in the other. That is the whole failure in one sentence: an ID is a pointer into one registry's address space, dereferenced against a different address space.

This also explains the sibling causes that produce the same 404 family:

• Registry state was lost. If the _schemas topic was deleted, retention-expired (it must be cleanup.policy=compact, never delete), or the registry was re-pointed at a fresh topic, previously issued IDs vanish. Records already on your business topics still reference them → 40403 (see schema-registry#667).
• The schema/version was hard-deleted. A soft delete (DELETE /subjects/{subject}/versions/{v}) leaves the ID resolvable; a hard delete (?permanent=true) removes it. A consumer replaying old offsets then hits 40403/40402.
• Subject-naming-strategy mismatch. With ID-based lookups this is rare, but paths that resolve by subject (use.latest.version=true, latest.compatibility.strict, some Protobuf reference resolution) call GET /subjects/{subject}/versions/.... If your producer used RecordNameStrategy (com.example.orders.OrderCreated) and your consumer expects TopicNameStrategy (orders-value), the subject doesn't exist → 40401.

5. The Fix

First, prove which registry has the ID. Read the actual ID off the wire and query both registries — do not guess.

# List everything the consumer's registry knows
curl -s http://localhost:8082/subjects
curl -s http://localhost:8082/schemas/ids/1     # the ID from the failing record

# Compare against the producer's registry
curl -s http://localhost:8081/schemas/ids/1

Whichever registry returns 200 for that ID is the one the consumer must point at.

Fix A — point the consumer at the correct registry (the 95% case)

The consumer's schema.registry.url was wrong. Fix the config, not the data.

- c.put("schema.registry.url", "http://localhost:8082");
+ c.put("schema.registry.url", "http://localhost:8081");

In Docker/K8s this is usually an address-reachability bug, not a typo: the consumer resolves a registry hostname that is a different instance from the producer's (per-namespace registries, a stale SCHEMA_REGISTRY_URL env var, a copy-pasted staging value in a dev pod). Treat the registry URL as a per-environment contract and inject it, never hardcode it.

Fix B — the schema is genuinely missing: re-register or restore

If the ID doesn't exist in any live registry, the registry lost state. Do not "fix" this by mutating consumers — restore the source of truth:

# Re-register the schema for the subject (gets a NEW id; only sound if IDs weren't
# already baked into records you must still read)
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"OrderCreated\",\"namespace\":\"com.example.orders\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"long\"}]}"}' \
  http://localhost:8081/subjects/orders-value/versions

Re-registering mints a new ID, which does not heal records already carrying the old ID. If historical records reference IDs from a wiped registry, the only real recovery is restoring the _schemas topic (from backup / MirrorMaker replica) so the original IDs resolve again. This is why the registry's storage topic must be compact, replicated RF=3, and backed up.

Fix C — subject-strategy or version mismatch (40401/40402)

Align the consumer's subject-name strategy and lookup mode with the producer's:

- props.put("value.subject.name.strategy", RecordNameStrategy.class.getName());
+ props.put("value.subject.name.strategy", TopicNameStrategy.class.getName()); // match producer

If you deliberately use use.latest.version=true, make sure the subject actually exists in the consumer's registry; that mode does a subject lookup, not an ID lookup, and is a common source of 40401.

Always wrap the deserializer so one bad ID doesn't crash-loop

Whatever the root cause, an uncaught SerializationException fires inside poll(), before your listener, so the container redelivers the same offset forever. Wrap the delegate:

// Spring Kafka: never let a registry lookup kill the consumer thread
props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, KafkaAvroDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(0L, 0L));
    // registry-not-found is not transient payload corruption; don't spin retries
    handler.addNotRetryableExceptions(SerializationException.class);
    return handler;
}

A word of caution: ErrorHandlingDeserializer + DLT is the right containment for a genuine poison pill, but a misconfigured registry URL is not a poison pill — it means every record routes to the DLT. Fix the config; use the wrapper so the consumer stays alive and you can see the DLT filling up instead of watching a crash loop.

6. Best Practices & The Better Design

The design that removes this class of bug is: one authoritative registry per data domain, injected as configuration, verified at startup.

• Producer and consumer share a registry identity. Global IDs are only portable within a registry, so both sides must resolve the same one. In multi-region setups, replicate schemas with Schema Linking (or MirrorMaker 2 including _schemas) using ID preservation, so an ID minted in region A resolves in region B. Never let two clusters mint IDs independently and expect cross-cluster reads to work.
• Registry URL is environment config, not code. Inject it (SCHEMA_REGISTRY_URL), and fail fast if it is unset:

String url = System.getenv("SCHEMA_REGISTRY_URL");
if (url == null || url.isBlank())
    throw new IllegalStateException("SCHEMA_REGISTRY_URL is required");
props.put("schema.registry.url", url);

• Pin the subject-naming strategy team-wide. Pick TopicNameStrategy (default) or RecordNameStrategy and enforce it in shared config, so producer and consumer never disagree on the subject key.
• Treat the registry as tier-1 stateful infrastructure. _schemas on cleanup.policy=compact, RF=3, min.insync.replicas=2, with backups. Losing it silently orphans every ID on every topic.
• Consumers can bootstrap their own schema cache from the correct source:

props.put("schema.registry.url", "http://schema-registry.prod.svc:8081");
props.put("auto.register.schemas", false);   // consumers never register
props.put("use.latest.version", false);      // resolve by wire ID, the safe default

7. How to Prevent It Long-Term

• Startup reachability probe. Before entering the poll loop, hit GET /subjects (or /schemas/ids/{knownId}) and fail deployment if the registry is unreachable or missing an expected subject — turn a runtime crash loop into a failed rollout.
• CI round-trip test with a real registry. Testcontainers Kafka + cp-schema-registry: produce with the production serializer config, consume with the production deserializer config, assert the record deserializes. Mocks never catch a wrong-registry or wrong-strategy bug — only an end-to-end round trip does.
• Alert on the exception and on DLT throughput. Any SerializationException with a 40403/40401/40402 cause is signal; a spike means an environment is pointed at the wrong registry or a schema was deleted. Alert on DLT produce rate and on records-consumed-rate dropping to zero while lag climbs.
• Guard the registry store. Alert if _schemas is not compact, if its RF drops below 3, or if any hard delete runs against a production subject. Back it up and rehearse restore.
• Config-as-code for registry URLs and strategies. Lint that no service hardcodes a registry host, and diff the effective serializer/deserializer config across environments in CI so a staging URL can't ride into a dev or prod manifest.

8. Key Takeaways

• SerializationException: Error retrieving Avro schema for id N + Schema not found; error code: 40403 means the consumer is asking a registry that does not have that ID — not a bad payload and not a compatibility break.
• The wire format carries a 4-byte global ID, and IDs are only portable within the registry that minted them. The usual root cause is a consumer pointed at a different registry than the producer.
• Diagnose with curl /schemas/ids/{id} against each registry; whichever returns 200 is the one the consumer must use. Fix the config, not the data.
• 40401 (Subject not found) and 40402 (Version not found) are the subject-lookup and hard-delete siblings — check subject-naming strategy and deletions.
• Wrap the deserializer in ErrorHandlingDeserializer so a lookup failure can't crash-loop the consumer, inject the registry URL per environment, and protect _schemas as tier-1 state.