> ## Content Index
> Fetch the complete content index at: https://www.ggorantala.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# Kafka SerializationException: Schema Not Found (40403) Fix
- URL: https://www.ggorantala.dev/kafka-schema-registry-40403-schema-not-found-fix/
- Published: 2026-05-04T13:32:00.000Z
- Updated: 2026-08-29T08:25:14.000Z
- Description: Kafka consumer dies with SerializationException: Error retrieving Avro schema for id N, Schema not found; error code: 40403. Why the ID lookup fails and how to fix it.
- Author: Gopi Gorantala
- Tags: kafka-errors, schema-registry, serializationexception, RestClientException, avro, spring-kafka, event-streaming

Your consumer was fine yesterday. Today, in a different environment, every record you poll blows up with a `SerializationException` complaining about a schema ID that plainly exists — you can see it in your other cluster. The consumer is in a crash loop, offsets aren't advancing, and lag is climbing.

This is not a compatibility problem and not a poison-pill payload. The 5-byte Confluent wire header on each record carries a schema **ID**, the deserializer takes that ID to a Schema Registry, and the registry says it has never heard of it. Almost always, the consumer is talking to a *different* registry than the one the producer registered against.

## 1\. The Error

The canonical stack trace on the consumer, wrapped up through `poll()`:

```text
org.apache.kafka.common.errors.SerializationException: Error retrieving Avro schema for id 42
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
    Schema not found; error code: 40403
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:308)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:395)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:788)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaByIdFromRegistry(CachedSchemaRegistryClient.java:302)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaBySubjectAndId(CachedSchemaRegistryClient.java:412)
    at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:...)

```

You will see one of a small family of REST error codes, and the distinction matters:

- `Schema not found; error code: 40403` — the deserializer asked `GET /schemas/ids/{id}` and no schema with that global ID exists in this registry.
- `Subject not found; error code: 40401` — a lookup keyed by subject (`GET /subjects/{subject}/versions/{version}`) found no such subject. Common with `use.latest.version=true`, ID-less lookups, or a subject-naming-strategy mismatch.
- `Version not found; error code: 40402` — the subject exists but the requested version doesn't (usually a hard-deleted version).

In Spring Kafka the same failure arrives wrapped one more time — `ListenerExecutionFailedException` on top, or, because deserialization happens inside `poll()` before your listener runs, as a container-level error that redelivers the same offset forever.

**Where it shows up:** Kafka `3.9`/`4.0` brokers, `kafka-avro-serializer` / `kafka-schema-registry-client` in the `7.x`–`8.2.x` line, Confluent Platform `7.x`–`8.x` or Confluent Cloud. The mechanics are identical for the Protobuf and JSON Schema deserializers — only the exception message noun changes.

## 2\. How to Reproduce It

Stand up a broker, a registry, produce one record so a schema gets registered, then point a consumer at a *second, empty* registry — exactly what happens when dev and staging each run their own.

```yaml
# docker-compose.yml — Kafka 3.9 KRaft + two Schema Registries
services:
  kafka:
    image: apache/kafka:3.9.0
    ports: ["9092:9092", "29092:29092"]
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://kafka:9092,CONTROLLER://kafka:9093,EXTERNAL://0.0.0.0:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  schema-registry-a:
    image: confluentinc/cp-schema-registry:7.9.0
    depends_on: [kafka]
    ports: ["8081:8081"]
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry-a
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_KAFKASTORE_TOPIC: _schemas_a

  schema-registry-b:
    image: confluentinc/cp-schema-registry:7.9.0
    depends_on: [kafka]
    ports: ["8082:8081"]
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry-b
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_KAFKASTORE_TOPIC: _schemas_b

```

Produce a record against registry **A** (port 8081):

```java
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
p.put("schema.registry.url", "http://localhost:8081"); // registry A

Schema schema = SchemaBuilder.record("OrderCreated").namespace("com.example.orders")
        .fields().requiredString("orderId").requiredLong("amount").endRecord();
GenericRecord rec = new GenericData.Record(schema);
rec.put("orderId", "o-1"); rec.put("amount", 4200L);

try (var producer = new KafkaProducer<String, GenericRecord>(p)) {
    producer.send(new ProducerRecord<>("orders", "o-1", rec)).get();
}

```

Registry A now holds schema ID 1\. Confirm it:

```bash
curl -s http://localhost:8081/schemas/ids/1        # 200, returns the schema
curl -s http://localhost:8082/schemas/ids/1        # {"error_code":40403,"message":"Schema not found"}

```

Now consume, pointing at registry **B** (port 8082), which is empty:

```java
Properties c = new Properties();
c.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
c.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-consumer");
c.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
c.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
c.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
c.put("schema.registry.url", "http://localhost:8082"); // registry B — WRONG

try (var consumer = new KafkaConsumer<String, GenericRecord>(c)) {
    consumer.subscribe(List.of("orders"));
    while (true) {
        var records = consumer.poll(Duration.ofMillis(500)); // throws here
        records.forEach(r -> System.out.println(r.value()));
    }
}

```

`poll()` throws `SerializationException: Error retrieving Avro schema for id 1 ... Schema not found; error code: 40403`. Same broker, same topic, same bytes — the only variable is which registry the consumer resolves the ID against.

## 3\. Why It Happens — Surface Level

The Confluent wire format does not carry the schema. It carries a **reference** to it: 1 magic byte (`0x0`) plus a 4-byte big-endian **global schema ID**. On deserialize, the client reads that ID and calls its configured Schema Registry — `GET /schemas/ids/{id}` — to fetch the writer schema. If the registry it is configured with does not contain that ID, you get a 404 → `40403`, wrapped as `SerializationException`.

The ID is only meaningful within the registry that minted it. Global IDs are **not portable across registries**. Producer registered ID 42 in registry A; your consumer is asking registry B, which assigns IDs from its own sequence and has no entry 42\. The bytes on the topic are perfectly valid; the lookup target is wrong.

## 4\. Why It Happens — Under the Hood

`AbstractKafkaAvroDeserializer.deserialize()` pulls the ID off the buffer and hands it to a `CachedSchemaRegistryClient`. That client keeps an in-memory `idToSchemaCache`; on a miss it issues `getSchemaByIdFromRegistry(id)` → `RestService.getId(id)` → `GET /schemas/ids/{id}`. A non-2xx REST response is thrown as `RestClientException` carrying the body's `error_code`, and the deserializer rethrows it as `SerializationException`. Because the ID isn't cached, **every** record with that ID re-hits the registry and re-throws — there is no negative caching of "not found."

Global IDs come from the registry's own allocator. Under the default `_schemas` Kafka-backed store, the registry is a compacted-topic-backed KV store; the ID sequence is derived from that log. Two independent registries (separate `_schemas` topics, or separate clusters) have **independent** sequences, so the *same* logical schema can be ID 1 in one and ID 7 in another, and an ID from one is simply absent in the other. That is the whole failure in one sentence: an ID is a pointer into one registry's address space, dereferenced against a different address space.

This also explains the sibling causes that produce the same 404 family:

- **Registry state was lost.** If the `_schemas` topic was deleted, retention-expired (it must be `cleanup.policy=compact`, never `delete`), or the registry was re-pointed at a fresh topic, previously issued IDs vanish. Records already on your business topics still reference them → `40403` (see [schema-registry#667](https://github.com/confluentinc/schema-registry/issues/667)).
- **The schema/version was hard-deleted.** A soft delete (`DELETE /subjects/{subject}/versions/{v}`) leaves the ID resolvable; a hard delete (`?permanent=true`) removes it. A consumer replaying old offsets then hits `40403`/`40402`.
- **Subject-naming-strategy mismatch.** With ID-based lookups this is rare, but paths that resolve by subject (`use.latest.version=true`, `latest.compatibility.strict`, some Protobuf reference resolution) call `GET /subjects/{subject}/versions/...`. If your producer used `RecordNameStrategy` (`com.example.orders.OrderCreated`) and your consumer expects `TopicNameStrategy` (`orders-value`), the subject doesn't exist → `40401`.

## 5\. The Fix

**First, prove which registry has the ID.** Read the actual ID off the wire and query both registries — do not guess.

```bash
# List everything the consumer's registry knows
curl -s http://localhost:8082/subjects
curl -s http://localhost:8082/schemas/ids/1     # the ID from the failing record

# Compare against the producer's registry
curl -s http://localhost:8081/schemas/ids/1

```

Whichever registry returns `200` for that ID is the one the consumer must point at.

### Fix A — point the consumer at the correct registry (the 95% case)

The consumer's `schema.registry.url` was wrong. Fix the config, not the data.

```diff
- c.put("schema.registry.url", "http://localhost:8082");
+ c.put("schema.registry.url", "http://localhost:8081");

```

In Docker/K8s this is usually an address-reachability bug, not a typo: the consumer resolves a registry hostname that is a *different instance* from the producer's (per-namespace registries, a stale `SCHEMA_REGISTRY_URL` env var, a copy-pasted staging value in a dev pod). Treat the registry URL as a per-environment contract and inject it, never hardcode it.

### Fix B — the schema is genuinely missing: re-register or restore

If the ID doesn't exist in *any* live registry, the registry lost state. Do not "fix" this by mutating consumers — restore the source of truth:

```bash
# Re-register the schema for the subject (gets a NEW id; only sound if IDs weren't
# already baked into records you must still read)
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"OrderCreated\",\"namespace\":\"com.example.orders\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"long\"}]}"}' \
  http://localhost:8081/subjects/orders-value/versions

```

Re-registering mints a *new* ID, which does **not** heal records already carrying the old ID. If historical records reference IDs from a wiped registry, the only real recovery is restoring the `_schemas` topic (from backup / MirrorMaker replica) so the original IDs resolve again. This is why the registry's storage topic must be `compact`, replicated `RF=3`, and backed up.

### Fix C — subject-strategy or version mismatch (`40401`/`40402`)

Align the consumer's subject-name strategy and lookup mode with the producer's:

```diff
- props.put("value.subject.name.strategy", RecordNameStrategy.class.getName());
+ props.put("value.subject.name.strategy", TopicNameStrategy.class.getName()); // match producer

```

If you deliberately use `use.latest.version=true`, make sure the subject actually exists in the consumer's registry; that mode does a subject lookup, not an ID lookup, and is a common source of `40401`.

### Always wrap the deserializer so one bad ID doesn't crash-loop

Whatever the root cause, an uncaught `SerializationException` fires **inside `poll()`**, before your listener, so the container redelivers the same offset forever. Wrap the delegate:

```java
// Spring Kafka: never let a registry lookup kill the consumer thread
props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, KafkaAvroDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);

```

```java
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(0L, 0L));
    // registry-not-found is not transient payload corruption; don't spin retries
    handler.addNotRetryableExceptions(SerializationException.class);
    return handler;
}

```

A word of caution: `ErrorHandlingDeserializer` \+ DLT is the right containment for a genuine poison pill, but a **misconfigured registry URL is not a poison pill** — it means *every* record routes to the DLT. Fix the config; use the wrapper so the consumer stays alive and you can see the DLT filling up instead of watching a crash loop.

## 6\. Best Practices & The Better Design

The design that removes this class of bug is: **one authoritative registry per data domain, injected as configuration, verified at startup.**

- **Producer and consumer share a registry identity.** Global IDs are only portable within a registry, so both sides must resolve the same one. In multi-region setups, replicate schemas with **Schema Linking** (or MirrorMaker 2 including `_schemas`) using ID preservation, so an ID minted in region A resolves in region B. Never let two clusters mint IDs independently and expect cross-cluster reads to work.
- **Registry URL is environment config, not code.** Inject it (`SCHEMA_REGISTRY_URL`), and fail fast if it is unset:

```java
String url = System.getenv("SCHEMA_REGISTRY_URL");
if (url == null || url.isBlank())
    throw new IllegalStateException("SCHEMA_REGISTRY_URL is required");
props.put("schema.registry.url", url);

```

- **Pin the subject-naming strategy team-wide.** Pick `TopicNameStrategy` (default) or `RecordNameStrategy` and enforce it in shared config, so producer and consumer never disagree on the subject key.
- **Treat the registry as tier-1 stateful infrastructure.** `_schemas` on `cleanup.policy=compact`, `RF=3`, `min.insync.replicas=2`, with backups. Losing it silently orphans every ID on every topic.
- **Consumers can bootstrap their own schema cache from the correct source:**

```java
props.put("schema.registry.url", "http://schema-registry.prod.svc:8081");
props.put("auto.register.schemas", false);   // consumers never register
props.put("use.latest.version", false);      // resolve by wire ID, the safe default

```

## 7\. How to Prevent It Long-Term

- **Startup reachability probe.** Before entering the poll loop, hit `GET /subjects` (or `/schemas/ids/{knownId}`) and fail deployment if the registry is unreachable or missing an expected subject — turn a runtime crash loop into a failed rollout.
- **CI round-trip test with a real registry.** `Testcontainers` Kafka + `cp-schema-registry`: produce with the production serializer config, consume with the production deserializer config, assert the record deserializes. Mocks never catch a wrong-registry or wrong-strategy bug — only an end-to-end round trip does.
- **Alert on the exception and on DLT throughput.** Any `SerializationException` with a `40403`/`40401`/`40402` cause is signal; a spike means an environment is pointed at the wrong registry or a schema was deleted. Alert on DLT produce rate and on `records-consumed-rate` dropping to zero while lag climbs.
- **Guard the registry store.** Alert if `_schemas` is not `compact`, if its RF drops below 3, or if any hard delete runs against a production subject. Back it up and rehearse restore.
- **Config-as-code for registry URLs and strategies.** Lint that no service hardcodes a registry host, and diff the effective serializer/deserializer config across environments in CI so a staging URL can't ride into a dev or prod manifest.

## 8\. Key Takeaways

- `SerializationException: Error retrieving Avro schema for id N` \+ `Schema not found; error code: 40403` means the consumer is asking a registry that does not have that ID — **not** a bad payload and **not** a compatibility break.
- The wire format carries a 4-byte global ID, and **IDs are only portable within the registry that minted them.** The usual root cause is a consumer pointed at a different registry than the producer.
- Diagnose with `curl /schemas/ids/{id}` against each registry; whichever returns `200` is the one the consumer must use. Fix the config, not the data.
- `40401` (Subject not found) and `40402` (Version not found) are the subject-lookup and hard-delete siblings — check subject-naming strategy and deletions.
- Wrap the deserializer in `ErrorHandlingDeserializer` so a lookup failure can't crash-loop the consumer, inject the registry URL per environment, and protect `_schemas` as tier-1 state.