Schema Being Registered Is Incompatible: Kafka 409 Fix

Your producer worked yesterday. You added one field to the Avro schema, redeployed, and now every send() fails with:

org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"OrderCreated","namespace":"com.acme.orders","fields":[{"name":"orderId","type":"string"},{"name":"amount","type":"double"},{"name":"currency","type":"string"}]}
	at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:130)
	at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59)
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1000)
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema for subject "orders-value", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field 'currency' at path '/fields/2' in the new schema has no default value and is missing in the old schema', additionalInfo:'currency'}, {oldSchemaVersion: 1}, {oldSchema: '...'}, {validateFields: 'false', compatibility: 'BACKWARD'}]; error code: 409
	at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:314)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:495)

On Schema Registry versions before 6.1 the message is just Schema being registered is incompatible with an earlier schema; error code: 409 with no details — same failure, less help.

Typical setup: kafka-clients 3.x, kafka-avro-serializer 7.x (Confluent Platform 7.x), Schema Registry running locally in Docker or as Confluent Cloud's managed registry. Spring Kafka surfaces the identical RestClientException wrapped in a SerializationException from the producer, and with JSON Schema or Protobuf serializers you get the same 409 with format-specific details.

Two things to internalize immediately: this error is produced by the Schema Registry, not by Kafka brokers — the broker never sees your message. And it is fatal for that record: the serializer fails before anything hits the wire, so every send with the new schema fails until you resolve it.

2. How to Reproduce It (step-by-step)

Minimal KRaft cluster plus Schema Registry:

# docker-compose.yml
services:
  kafka:
    image: confluentinc/cp-kafka:7.6.1
    ports: ["9092:9092"]
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk

  schema-registry:
    image: confluentinc/cp-schema-registry:7.6.1
    depends_on: [kafka]
    ports: ["8081:8081"]
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

Register schema v1 for subject orders-value (two fields):

curl -s -X POST http://localhost:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema":"{\"type\":\"record\",\"name\":\"OrderCreated\",\"namespace\":\"com.acme.orders\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"}'
# {"id":1}

Now register v2, adding currency as a required field without a default — the single most common way teams hit this:

curl -s -X POST http://localhost:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema":"{\"type\":\"record\",\"name\":\"OrderCreated\",\"namespace\":\"com.acme.orders\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}"}'
# {"error_code":409,"message":"Schema being registered is incompatible with an earlier schema for subject \"orders-value\" ..."}

The same failure fires from a producer without any curl involved, because KafkaAvroSerializer auto-registers by default:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
          io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://localhost:8081");
// auto.register.schemas defaults to true — the serializer will attempt
// to register the v2 schema on first send and take the 409 in the face.

try (KafkaProducer<String, GenericRecord> producer = new KafkaProducer<>(props)) {
    producer.send(new ProducerRecord<>("orders", "o-1", recordWithCurrencyField)).get();
}

Environment-specific trigger worth knowing: this only bites when the subject already has a version. Fresh local environments (wiped volumes, new Confluent Cloud env) register v1 cleanly, which is why the breaking change sails through dev and detonates in staging or prod where v1 already exists.

3. Why It Happens — Surface Level

Every subject in Schema Registry has a compatibility level. The default is BACKWARD: any new schema version must be able to read data written with the previous version. When the serializer (or your curl) POSTs a new schema, the registry diffs it against the latest registered version under that rule. Your new schema declares currency as required, but v1 data contains no currency field and there's no default to fall back on — a consumer using the new schema could not decode old records. The registry rejects the registration with HTTP 409, the serializer wraps it in SerializationException, and your send fails.

It's not a Kafka error, not a network error, not a serialization bug. It's the registry doing exactly its job: refusing a schema evolution that would break somebody.

4. Why It Happens — Under the Hood

The registration path: KafkaAvroSerializer.serialize() → CachedSchemaRegistryClient.register() → POST /subjects/{subject}/versions. The subject name comes from the subject naming strategy, default TopicNameStrategy, i.e. <topic>-value for values and <topic>-key for keys. Compatibility is enforced per subject, so the same schema can be legal on one topic and rejected on another.

Server-side, the registry loads the subject's version history from its internal _schemas compacted topic (the registry is itself just a Kafka consumer with an in-memory index) and runs the Avro resolution rules between reader and writer schemas. What each level actually checks:

• BACKWARD (default): new schema reads old data. Legal: delete fields, add fields with defaults, widen types per Avro promotion rules. Illegal: add required fields, rename without alias, narrow types.
• FORWARD: old schema reads new data. Legal: add fields, delete fields that had defaults.
• FULL: both directions. Effectively: only add/remove fields with defaults.
• *_TRANSITIVE variants: checked against all previous versions, not just the latest. BACKWARD (non-transitive) only compares against the latest version — which means a sequence of individually-compatible changes can still leave v3 unable to read v1 data. Transitive closes that hole.
• NONE: no check. Anything registers. (This is how poison schemas get in — see the poison-pill article for what happens on the consumer side.)

Two details that regularly confuse people during the incident:

The check direction is about consumers, not producers. BACKWARD exists so you can upgrade consumers first: a consumer compiled against v2 must handle the v1 records still sitting in the topic (retention!). The registry can't know your deployment order, so it enforces the contract at registration time.

The 409 can appear without any schema "change" you made. If two services produce to the same topic with slightly different generated classes (different code-gen versions emitting different doc/order/logical types, or one service missing a field), the second service's register call is a new-schema registration attempt and gets compatibility-checked. Same 409, no intentional evolution anywhere. TopicNameStrategy + multiple producer codebases is the usual culprit.

5. The Fix

Fix 1: Make the evolution compatible (correct answer 95% of the time)

Give the new field a default so new readers can decode old records:

 {
   "type": "record",
   "name": "OrderCreated",
   "namespace": "com.acme.orders",
   "fields": [
     {"name": "orderId", "type": "string"},
     {"name": "amount", "type": "double"},
-    {"name": "currency", "type": "string"}
+    {"name": "currency", "type": ["null", "string"], "default": null}
   ]
 }

Or, if a sensible sentinel exists: {"name": "currency", "type": "string", "default": "EUR"}. The nullable-union form is the standard pattern; note the default must match the first branch of the union (null first, "default": null).

Verify before you deploy — the registry has a dry-run endpoint:

curl -s -X POST \
  http://localhost:8081/compatibility/subjects/orders-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d @new-schema.json
# {"is_compatible":true}

Fix 2: The break is intentional — you really want a required field

Then this is a new contract, not an evolution. Options in order of preference:

1. New topic (orders.v2) with its own subject. Clean cut, consumers migrate on their own schedule. This is the fintech-grade answer for genuinely breaking changes.
2. Relax the subject's compatibility — temporarily and deliberately:

curl -s -X PUT http://localhost:8081/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "NONE"}'
# register the breaking schema, then restore:
curl -s -X PUT http://localhost:8081/config/orders-value \
  -d '{"compatibility": "BACKWARD"}'

Understand what you just did: any consumer reading with the new schema will throw on every pre-change record still within retention. Only acceptable if you control all consumers and drain/reset them in lockstep.

3. Delete the offending subject versions (dev/staging reset, never prod):

curl -s -X DELETE http://localhost:8081/subjects/orders-value   # soft delete
curl -s -X DELETE "http://localhost:8081/subjects/orders-value?permanent=true"

Fix 3: You never meant to register from the producer at all

If the 409 came from an app auto-registering a slightly-different generated schema (the multi-producer case from section 4), stop letting apps register:

 props.put("schema.registry.url", "http://localhost:8081");
-# auto.register.schemas defaults to true
+props.put("auto.register.schemas", "false");
+props.put("use.latest.version", "true");

With auto.register.schemas=false the serializer only looks up the schema; with use.latest.version=true it serializes using the latest registered version instead of demanding a byte-identical match (avoiding the follow-up Schema not found; error code: 40403). Registration becomes a controlled pipeline step, not a side effect of send().

6. Best Practices & The Better Design

The class of problem here is "schema changes discovered at runtime." The better design moves the compatibility check to build time and strips production apps of registration rights.

Schemas live in the repo, and CI runs the same check the registry would, via the kafka-schema-registry-maven-plugin:

<plugin>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-schema-registry-maven-plugin</artifactId>
  <version>7.6.1</version>
  <configuration>
    <schemaRegistryUrls>
      <param>https://schema-registry.internal:8081</param>
    </schemaRegistryUrls>
    <subjects>
      <orders-value>src/main/avro/OrderCreated.avsc</orders-value>
    </subjects>
  </configuration>
</plugin>

mvn schema-registry:test-compatibility   # fails the build instead of prod
mvn schema-registry:register             # CD pipeline registers, apps don't

Production serializer config, the right way:

# producer
schema.registry.url=https://schema-registry.internal:8081
auto.register.schemas=false
use.latest.version=true
latest.compatibility.strict=true

And for topics where multiple event types or long retention matter, set the subject to transitive checking:

curl -s -X PUT https://schema-registry.internal:8081/config/orders-value \
  -d '{"compatibility": "FULL_TRANSITIVE"}'

FULL_TRANSITIVE on business-critical subjects means field additions/removals always carry defaults, so producers and consumers deploy in any order against any retained data. That single convention eliminates the entire "who upgrades first" negotiation.

Pair this with the consumer-side story: ErrorHandlingDeserializer + DLT protects consumers from bad bytes; build-time compatibility testing protects them from bad contracts. You want both.

7. How to Prevent It Long-Term

Make test-compatibility a required CI check on every PR that touches an .avsc/.proto/JSON-schema file — this converts the 409 from a production incident into a red pull request. Lock down who can register: in Confluent Cloud, give applications DeveloperRead on subjects and reserve write to the CI service account; on self-hosted, front the registry with network policy so only the deploy pipeline reaches POST /subjects/*. Set an explicit compatibility level per subject in code (Terraform's confluent_subject_config or a bootstrap script) rather than inheriting the global default silently — an unreviewed global NONE is how incompatible schemas end up registered and consumers start dying weeks later. Alert on registry 4xx rates and on FailedProduceRequestsPerSec-adjacent app metrics: a spike in producer SerializationException is a deploy gone wrong, and you want the page before the retry queue fills. Finally, agree on evolution rules as team convention: new fields always nullable-with-default, never rename (use aliases), never retype, breaking change = new topic.

8. Key Takeaways

• Error code 409 is the Schema Registry rejecting registration — the broker never saw your record, and the failure is fatal per-send until the schema conflict is resolved.
• Default compatibility is BACKWARD: new schemas must read old data. Adding a required field without a default is the #1 trigger; fix it with ["null","string"] + "default": null.
• BACKWARD only checks against the latest version — use FULL_TRANSITIVE on critical subjects to guard all retained data.
• In production set auto.register.schemas=false and use.latest.version=true; registration belongs in CI/CD, not in send().
• POST /compatibility/subjects/{subject}/versions/latest is a free dry-run — wire it into CI via mvn schema-registry:test-compatibility and the 409 becomes a failed build, not an incident.