> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Schema Being Registered Is Incompatible: Kafka 409 Fix - URL: https://www.ggorantala.dev/kafka-schema-registry-409-incompatible-schema/ - Published: 2026-05-12T13:29:00.000Z - Updated: 2026-08-29T08:25:21.000Z - Description: Fix "Schema being registered is incompatible with an earlier schema; error code: 409" — why Schema Registry rejects your schema and how to evolve safely. - Author: Gopi Gorantala - Tags: kafka-errors, schema-registry, avro, serializationexception, schema-evolution, srping-kaf, Java Your producer worked yesterday. You added one field to the Avro schema, redeployed, and now every `send()` fails with: ```text org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"OrderCreated","namespace":"com.acme.orders","fields":[{"name":"orderId","type":"string"},{"name":"amount","type":"double"},{"name":"currency","type":"string"}]} at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:130) at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1000) Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema for subject "orders-value", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field 'currency' at path '/fields/2' in the new schema has no default value and is missing in the old schema', additionalInfo:'currency'}, {oldSchemaVersion: 1}, {oldSchema: '...'}, {validateFields: 'false', compatibility: 'BACKWARD'}]; error code: 409 at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:314) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:495) ``` On Schema Registry versions before 6.1 the message is just `Schema being registered is incompatible with an earlier schema; error code: 409` with no details — same failure, less help. Typical setup: kafka-clients 3.x, `kafka-avro-serializer` 7.x (Confluent Platform 7.x), Schema Registry running locally in Docker or as Confluent Cloud's managed registry. Spring Kafka surfaces the identical `RestClientException` wrapped in a `SerializationException` from the producer, and with JSON Schema or Protobuf serializers you get the same 409 with format-specific `details`. Two things to internalize immediately: this error is produced by the **Schema Registry**, not by Kafka brokers — the broker never sees your message. And it is **fatal for that record**: the serializer fails before anything hits the wire, so every send with the new schema fails until you resolve it. ## 2\. How to Reproduce It (step-by-step) Minimal KRaft cluster plus Schema Registry: ```yaml # docker-compose.yml services: kafka: image: confluentinc/cp-kafka:7.6.1 ports: ["9092:9092"] environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093 KAFKA_LISTENERS: PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,EXTERNAL://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,EXTERNAL://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk schema-registry: image: confluentinc/cp-schema-registry:7.6.1 depends_on: [kafka] ports: ["8081:8081"] environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092 SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081 ``` Register schema v1 for subject `orders-value` (two fields): ```bash curl -s -X POST http://localhost:8081/subjects/orders-value/versions \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{"schema":"{\"type\":\"record\",\"name\":\"OrderCreated\",\"namespace\":\"com.acme.orders\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"}' # {"id":1} ``` Now register v2, adding `currency` as a **required field without a default** — the single most common way teams hit this: ```bash curl -s -X POST http://localhost:8081/subjects/orders-value/versions \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{"schema":"{\"type\":\"record\",\"name\":\"OrderCreated\",\"namespace\":\"com.acme.orders\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}"}' # {"error_code":409,"message":"Schema being registered is incompatible with an earlier schema for subject \"orders-value\" ..."} ``` The same failure fires from a producer without any curl involved, because `KafkaAvroSerializer` auto-registers by default: ```java Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class); props.put("schema.registry.url", "http://localhost:8081"); // auto.register.schemas defaults to true — the serializer will attempt // to register the v2 schema on first send and take the 409 in the face. try (KafkaProducer producer = new KafkaProducer<>(props)) { producer.send(new ProducerRecord<>("orders", "o-1", recordWithCurrencyField)).get(); } ``` Environment-specific trigger worth knowing: this only bites when the *subject already has a version*. Fresh local environments (wiped volumes, new Confluent Cloud env) register v1 cleanly, which is why the breaking change sails through dev and detonates in staging or prod where v1 already exists. ## 3\. Why It Happens — Surface Level Every subject in Schema Registry has a compatibility level. The default is `BACKWARD`: any new schema version must be able to **read data written with the previous version**. When the serializer (or your curl) POSTs a new schema, the registry diffs it against the latest registered version under that rule. Your new schema declares `currency` as required, but v1 data contains no `currency` field and there's no default to fall back on — a consumer using the new schema could not decode old records. The registry rejects the registration with HTTP 409, the serializer wraps it in `SerializationException`, and your send fails. It's not a Kafka error, not a network error, not a serialization bug. It's the registry doing exactly its job: refusing a schema evolution that would break somebody. ## 4\. Why It Happens — Under the Hood The registration path: `KafkaAvroSerializer.serialize()` → `CachedSchemaRegistryClient.register()` → `POST /subjects/{subject}/versions`. The subject name comes from the **subject naming strategy**, default `TopicNameStrategy`, i.e. `-value` for values and `-key` for keys. Compatibility is enforced **per subject**, so the same schema can be legal on one topic and rejected on another. Server-side, the registry loads the subject's version history from its internal `_schemas` compacted topic (the registry is itself just a Kafka consumer with an in-memory index) and runs the Avro resolution rules between reader and writer schemas. What each level actually checks: - `BACKWARD` (default): new schema reads old data. Legal: delete fields, add fields **with defaults**, widen types per Avro promotion rules. Illegal: add required fields, rename without alias, narrow types. - `FORWARD`: old schema reads new data. Legal: add fields, delete fields **that had defaults**. - `FULL`: both directions. Effectively: only add/remove fields with defaults. - `*_TRANSITIVE` variants: checked against **all** previous versions, not just the latest. `BACKWARD` (non-transitive) only compares against the latest version — which means a sequence of individually-compatible changes can still leave v3 unable to read v1 data. Transitive closes that hole. - `NONE`: no check. Anything registers. (This is how poison schemas get in — see the poison-pill article for what happens on the consumer side.) Two details that regularly confuse people during the incident: **The check direction is about consumers, not producers.** `BACKWARD` exists so you can upgrade consumers first: a consumer compiled against v2 must handle the v1 records still sitting in the topic (retention!). The registry can't know your deployment order, so it enforces the contract at registration time. **The 409 can appear without any schema "change" you made.** If two services produce to the same topic with slightly different generated classes (different code-gen versions emitting different doc/order/logical types, or one service missing a field), the second service's register call is a new-schema registration attempt and gets compatibility-checked. Same 409, no intentional evolution anywhere. `TopicNameStrategy` \+ multiple producer codebases is the usual culprit. ## 5\. The Fix ### Fix 1: Make the evolution compatible (correct answer 95% of the time) Give the new field a default so new readers can decode old records: ```diff { "type": "record", "name": "OrderCreated", "namespace": "com.acme.orders", "fields": [ {"name": "orderId", "type": "string"}, {"name": "amount", "type": "double"}, - {"name": "currency", "type": "string"} + {"name": "currency", "type": ["null", "string"], "default": null} ] } ``` Or, if a sensible sentinel exists: `{"name": "currency", "type": "string", "default": "EUR"}`. The nullable-union form is the standard pattern; note the default must match the **first** branch of the union (`null` first, `"default": null`). Verify before you deploy — the registry has a dry-run endpoint: ```bash curl -s -X POST \ http://localhost:8081/compatibility/subjects/orders-value/versions/latest \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d @new-schema.json # {"is_compatible":true} ``` ### Fix 2: The break is intentional — you really want a required field Then this is a new contract, not an evolution. Options in order of preference: 1. **New topic** (`orders.v2`) with its own subject. Clean cut, consumers migrate on their own schedule. This is the fintech-grade answer for genuinely breaking changes. 2. **Relax the subject's compatibility** — temporarily and deliberately: ```bash curl -s -X PUT http://localhost:8081/config/orders-value \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{"compatibility": "NONE"}' # register the breaking schema, then restore: curl -s -X PUT http://localhost:8081/config/orders-value \ -d '{"compatibility": "BACKWARD"}' ``` Understand what you just did: any consumer reading with the new schema will throw on every pre-change record still within retention. Only acceptable if you control all consumers and drain/reset them in lockstep. 1. **Delete the offending subject versions** (dev/staging reset, never prod): ```bash curl -s -X DELETE http://localhost:8081/subjects/orders-value # soft delete curl -s -X DELETE "http://localhost:8081/subjects/orders-value?permanent=true" ``` ### Fix 3: You never meant to register from the producer at all If the 409 came from an app auto-registering a slightly-different generated schema (the multi-producer case from section 4), stop letting apps register: ```diff props.put("schema.registry.url", "http://localhost:8081"); -# auto.register.schemas defaults to true +props.put("auto.register.schemas", "false"); +props.put("use.latest.version", "true"); ``` With `auto.register.schemas=false` the serializer only **looks up** the schema; with `use.latest.version=true` it serializes using the latest registered version instead of demanding a byte-identical match (avoiding the follow-up `Schema not found; error code: 40403`). Registration becomes a controlled pipeline step, not a side effect of `send()`. ## 6\. Best Practices & The Better Design The class of problem here is "schema changes discovered at runtime." The better design moves the compatibility check to build time and strips production apps of registration rights. Schemas live in the repo, and CI runs the same check the registry would, via the `kafka-schema-registry-maven-plugin`: ```xml io.confluent kafka-schema-registry-maven-plugin 7.6.1 https://schema-registry.internal:8081 src/main/avro/OrderCreated.avsc ``` ```bash mvn schema-registry:test-compatibility # fails the build instead of prod mvn schema-registry:register # CD pipeline registers, apps don't ``` Production serializer config, the right way: ```properties # producer schema.registry.url=https://schema-registry.internal:8081 auto.register.schemas=false use.latest.version=true latest.compatibility.strict=true ``` And for topics where multiple event types or long retention matter, set the subject to transitive checking: ```bash curl -s -X PUT https://schema-registry.internal:8081/config/orders-value \ -d '{"compatibility": "FULL_TRANSITIVE"}' ``` `FULL_TRANSITIVE` on business-critical subjects means field additions/removals always carry defaults, so producers and consumers deploy in any order against any retained data. That single convention eliminates the entire "who upgrades first" negotiation. Pair this with the consumer-side story: `ErrorHandlingDeserializer` \+ DLT protects consumers from bad *bytes*; build-time compatibility testing protects them from bad *contracts*. You want both. ## 7\. How to Prevent It Long-Term Make `test-compatibility` a required CI check on every PR that touches an `.avsc`/`.proto`/JSON-schema file — this converts the 409 from a production incident into a red pull request. Lock down who can register: in Confluent Cloud, give applications `DeveloperRead` on subjects and reserve write to the CI service account; on self-hosted, front the registry with network policy so only the deploy pipeline reaches `POST /subjects/*`. Set an explicit compatibility level per subject in code (Terraform's `confluent_subject_config` or a bootstrap script) rather than inheriting the global default silently — an unreviewed global `NONE` is how incompatible schemas end up registered and consumers start dying weeks later. Alert on registry 4xx rates and on `FailedProduceRequestsPerSec`\-adjacent app metrics: a spike in producer `SerializationException` is a deploy gone wrong, and you want the page before the retry queue fills. Finally, agree on evolution rules as team convention: new fields always nullable-with-default, never rename (use aliases), never retype, breaking change = new topic. ## 8\. Key Takeaways - Error code 409 is the **Schema Registry** rejecting registration — the broker never saw your record, and the failure is fatal per-send until the schema conflict is resolved. - Default compatibility is `BACKWARD`: new schemas must read old data. Adding a required field without a default is the #1 trigger; fix it with `["null","string"]` \+ `"default": null`. - `BACKWARD` only checks against the *latest* version — use `FULL_TRANSITIVE` on critical subjects to guard all retained data. - In production set `auto.register.schemas=false` and `use.latest.version=true`; registration belongs in CI/CD, not in `send()`. - `POST /compatibility/subjects/{subject}/versions/latest` is a free dry-run — wire it into CI via `mvn schema-registry:test-compatibility` and the 409 becomes a failed build, not an incident.