Transactions in Spring Kafka

Many Kafka consumers do more than just read: they consume a record, process it, and produce one or more results to another topic. Without transactions, a crash between “produce result” and “commit offset” leaves you with duplicate output on redelivery, while a crash in the other order silently drops messages. Spring for Apache Kafka wraps Kafka’s native transactional API behind a KafkaTransactionManager so the produced records and the consumed offsets commit (or roll back) as a single atomic unit — the foundation of exactly-once read-process-write.

How Kafka transactions work

A transactional producer is identified by a stable transactional.id. On startup it registers with the transaction coordinator, fences any older producer instance sharing that id (preventing zombies), and begins emitting records inside transactions. Crucially, the consumer’s offsets can be committed through the same producer transaction via sendOffsetsToTransaction. Either everything in the transaction becomes visible to downstream read_committed consumers, or nothing does.

Spring automates all of this. When a transactional KafkaTemplate runs inside a Spring transaction managed by KafkaTransactionManager, the container’s consumer offsets are added to the same producer transaction automatically — you never call sendOffsetsToTransaction yourself.

Enabling transactions

Setting a transaction-id-prefix on the producer is the single switch that turns the auto-configured ProducerFactory transactional and registers a KafkaTransactionManager.

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      transaction-id-prefix: "orders-tx-"
      acks: all                         # required; transactions imply idempotence
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      group-id: order-processor
      isolation-level: read_committed    # only see committed records
      enable-auto-commit: false          # offsets commit via the transaction

The container appends a unique suffix to transaction-id-prefix per consumer thread, so the prefix only needs to be unique per application, not per instance. Two apps sharing a prefix will fence each other — give each deployment its own prefix.

A transactional listener

With the prefix set, annotate the listener method (or its enclosing service call) with @Transactional. Spring starts a Kafka transaction before the method runs; the KafkaTemplate.send inside it joins that transaction, and the consumed offset is committed atomically when the method returns.

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

public record OrderPlaced(String orderId, String sku, int qty) {}
public record ShipmentRequested(String orderId, String warehouse) {}

@Service
public class OrderProcessor {

    private final KafkaTemplate<String, Object> kafkaTemplate;

    public OrderProcessor(KafkaTemplate<String, Object> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    @KafkaListener(topics = "orders", groupId = "order-processor")
    @Transactional
    public void onOrder(OrderPlaced order) {
        var shipment = new ShipmentRequested(order.orderId(), pickWarehouse(order.sku()));
        // This send and the offset commit succeed or fail together.
        kafkaTemplate.send("shipments", order.orderId(), shipment);
    }

    private String pickWarehouse(String sku) {
        return sku.startsWith("EU-") ? "rotterdam" : "memphis";
    }
}

If onOrder throws, the Kafka transaction is rolled back: the shipments record is never made visible to read_committed consumers and the orders offset is not committed, so the record is redelivered.

Do not call KafkaTemplate.executeInTransaction(...) from inside a container-managed transaction. That starts a separate local transaction that is not synchronized with the consumer offsets, defeating exactly-once.

Chaining with a database transaction

The hard part is read-process-write-to-DB. You want the JPA insert and the Kafka publish to both commit or both roll back. Two records do not share a single atomic commit (Kafka and your database are different systems), so a true 2PC is impossible without a distributed transaction coordinator.

Historically people reached for ChainedTransactionManager, which nests two transaction managers so they commit in sequence:

// Deprecated since Spring 5.3 — best-effort, NOT atomic.
var chained = new ChainedKafkaTransactionManager<>(kafkaTxManager, jpaTxManager);

The caveat is fundamental: chaining is best-effort 1PC, not atomic. If the database commit succeeds but the subsequent Kafka commit fails (or the JVM dies in between), you get a row with no published event — a lost message. ChainedTransactionManager is deprecated for exactly this reason.

The transactional outbox (recommended)

The reliable pattern is the outbox: write the event into an outbox table in the same JPA transaction as your business data, then publish it to Kafka separately (via a relay/poller or change-data-capture like Debezium).

@Service
public class OrderService {

    private final OrderRepository orders;
    private final OutboxRepository outbox;

    public OrderService(OrderRepository orders, OutboxRepository outbox) {
        this.orders = orders;
        this.outbox = outbox;
    }

    @Transactional   // a single JPA transaction — atomic by construction
    public void place(OrderPlaced cmd) {
        orders.save(new OrderEntity(cmd.orderId(), cmd.sku(), cmd.qty()));
        outbox.save(OutboxEvent.of("shipments", cmd.orderId(), toJson(cmd)));
    }
}

A separate Kafka transaction publishes outbox rows and marks them sent. Because the write to the outbox is part of the DB commit, no event is ever lost; the relay provides at-least-once delivery, and consumers dedupe via the event id.

Approach	Atomic DB + Kafka	Operational cost	Verdict
Kafka-only `@Transactional`	N/A (no DB)	Low	Use when there is no DB write
`ChainedTransactionManager`	No (best-effort)	Low	Deprecated — avoid
Outbox + relay/CDC	Effectively yes	Medium	Recommended for DB + Kafka

Best Practices

Set a transaction-id-prefix that is unique per deployed application; never share it across instances or you will trigger producer fencing.
Always pair transactions with acks: all and consumers set to isolation-level: read_committed, otherwise readers see uncommitted records.
Keep transactional listeners short and side-effect-free apart from the sends — long DB calls inside a Kafka transaction hold the transaction open and risk transaction.timeout.ms expiry.
Prefer the outbox pattern over ChainedTransactionManager whenever a database write must be consistent with a Kafka publish.
Configure the DefaultAfterRollbackProcessor (the container default) so rolled-back records retry and eventually route to a dead-letter topic instead of looping forever.
Make downstream consumers idempotent; transactions give exactly-once within Kafka, but cross-system delivery is still effectively at-least-once.