Schema Evolution & Compatibility

Events in Kafka are long-lived: a topic retains messages for days, and consumers read at their own pace. That means a producer and a consumer almost never run the exact same schema version at the same time. Schema evolution is the discipline of changing your Avro, Protobuf, or JSON Schema definitions over time without breaking the applications that read the old or new data. The Schema Registry enforces this for you by rejecting any new schema version that would violate the compatibility rule configured for the subject.

Why compatibility matters

When a producer serializes a record, the registry assigns the schema a unique ID that is embedded in the message bytes. A consumer later fetches the schema by that ID and deserializes. If you deploy a new producer with a changed schema while old consumers are still running — or roll out new consumers against a backlog of old messages — the two schemas must be compatible or deserialization fails at runtime. Compatibility checks move that failure to deploy time, where you can catch it in CI instead of in production.

Compatibility types

Compatibility is evaluated between a new candidate schema and one or more previously registered versions of the same subject. The mode determines which direction must hold and how far back the check reaches.

Mode	Guarantee	Upgrade order
`BACKWARD`	New schema can read data written with the latest old schema	Upgrade consumers first
`BACKWARD_TRANSITIVE`	New schema can read data written with all previous schemas	Upgrade consumers first
`FORWARD`	Latest old schema can read data written with the new schema	Upgrade producers first
`FORWARD_TRANSITIVE`	All previous schemas can read data written with the new schema	Upgrade producers first
`FULL`	Both backward and forward against the latest version	Either order
`FULL_TRANSITIVE`	Both directions against all versions	Either order
`NONE`	No checks — anything is accepted	None (you own the risk)

BACKWARD is the registry default and the most common choice: it lets you upgrade consumers ahead of producers, which matches how most teams roll out releases.

What changes are safe

The rules follow directly from how Avro/Protobuf resolve a “reader” schema against a “writer” schema. The key insight: a consumer using the new schema must be able to fill in any field the old data did not contain, which is only possible if that field has a default.

Change	BACKWARD	FORWARD	FULL
Add a field with a default	Safe	Safe	Safe
Add a field without a default	Breaking	Safe	Breaking
Remove a field with a default	Safe	Safe	Safe
Remove a field without a default	Safe	Breaking	Breaking
Rename a field (no alias)	Breaking	Breaking	Breaking
Widen type (`int` → `long`)	Safe	Breaking	Breaking
Change a field’s default value	Safe	Safe	Safe
Add a value to an `enum`	Breaking*	Safe	Breaking

* Adding an enum symbol is backward-incompatible in Avro unless the enum declares a default.

Tip: If you make every field optional with a sensible default from day one, the vast majority of future changes stay FULL-compatible. Defaults are the single most important habit in schema design.

Setting compatibility per subject

Compatibility is configured globally and can be overridden per subject. The per-subject setting is what you usually want, so different topics can evolve at different speeds.

# Inspect the current global level
curl -s http://localhost:8081/config | jq

# Set the global default
curl -s -X PUT http://localhost:8081/config \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}'

# Override for one subject (topic "orders", value schema)
curl -s -X PUT http://localhost:8081/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL_TRANSITIVE"}'

Output:

{"compatibility":"FULL_TRANSITIVE"}

You can also test a candidate schema before registering it, which is ideal for a CI gate:

curl -s -X POST http://localhost:8081/compatibility/subjects/orders-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d @order-v2.json | jq

Output:

{"is_compatible": true}

A practical evolution example

Start with an OrderPlaced event. Version 1 carries the essentials:

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.devcraftly.orders",
  "fields": [
    { "name": "orderId", "type": "string" },
    { "name": "amount",  "type": "double" }
  ]
}

Later, the business needs a currency and an optional loyalty tier. Under BACKWARD compatibility, both new fields must have defaults so a new consumer can read v1 records that lack them:

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.devcraftly.orders",
  "fields": [
    { "name": "orderId",  "type": "string" },
    { "name": "amount",   "type": "double" },
    { "name": "currency", "type": "string", "default": "USD" },
    { "name": "tier",     "type": ["null", "string"], "default": null }
  ]
}

This schema registers successfully and the registry bumps it to version 2. Old consumers (still on v1) ignore the unknown fields; new consumers reading v1 messages see currency = "USD" and tier = null.

In Spring Boot the producer code is unchanged — you just publish the generated record and the Avro serializer handles registration and the schema ID:

@Service
public class OrderEventProducer {

    private final KafkaTemplate<String, OrderPlaced> kafkaTemplate;

    public OrderEventProducer(KafkaTemplate<String, OrderPlaced> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publish(OrderPlaced event) {
        kafkaTemplate.send("orders", event.getOrderId(), event);
    }
}

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
      properties:
        schema.registry.url: http://localhost:8081
        auto.register.schemas: false   # register via CI, not at runtime

Setting auto.register.schemas: false in production is deliberate: schemas should be registered by a controlled pipeline that runs the compatibility check, not silently by whichever producer starts first.

Best practices

Default to BACKWARD and upgrade consumers before producers; switch to FULL_TRANSITIVE for events shared across many independent teams.
Give every new field a default (null for optionals) — this keeps most changes compatible and removes the need for coordinated deploys.
Never rename or repurpose a field; add a new one and deprecate the old. Use Avro aliases if a rename is unavoidable.
Run the /compatibility check in CI so an incompatible change fails the build, never a running consumer.
Disable auto.register.schemas in production and register schemas through a reviewed pipeline.
Use the *_TRANSITIVE variants when consumers may replay old data from the start of a long-retention topic.