Delivery Semantics

Delivery semantics describe what happens to a message when something fails — a broker crashes, a producer retries, a consumer dies mid-processing. Kafka offers three guarantees: at-most-once, at-least-once, and exactly-once. The crucial thing to understand is that the end-to-end guarantee is not a single switch; it emerges from the combination of how the producer publishes (acknowledgements and idempotence) and how the consumer commits offsets relative to its processing. Getting this wrong is the root cause of most “we lost a message” and “we processed it twice” incidents in production.

The three semantics

At-most-once means a message is delivered zero or one times — duplicates are impossible, but messages can be lost. This happens when the producer does not wait for acknowledgement, or when the consumer commits offsets before it finishes processing. If the consumer crashes after committing but before the work is done, that record is silently skipped.

At-least-once means a message is delivered one or more times — nothing is lost, but duplicates are possible. The producer retries until it gets confirmation, and the consumer commits offsets only after processing succeeds. If a crash occurs between processing and committing, the record is re-delivered and reprocessed. This is the practical default for most Kafka applications.

Exactly-once means each message takes effect once and only once, even across retries and failures. In Kafka this is achieved with the idempotent producer plus transactions that atomically tie produced messages to consumed offsets (the read-process-write pattern). It is powerful but adds latency and operational complexity, so reach for it only when duplicate processing genuinely cannot be tolerated.

At-least-once is the right default for the vast majority of systems. Combine it with idempotent consumers (dedup keys, upserts) and you get effectively-once behaviour without the cost of Kafka transactions.

How producer and consumer settings combine

The semantic you actually get is the weaker of the two ends. A perfectly durable producer paired with a consumer that commits before processing still yields at-most-once. Use this table to reason about both sides together.

Semantic	Producer settings	Consumer settings
At-most-once	`acks=0` (fire-and-forget)	`enable.auto.commit=true`, or commit offsets before processing
At-least-once	`acks=all`, `retries` > 0, `enable.idempotence=true`	`enable.auto.commit=false`, commit after processing succeeds
Exactly-once	`enable.idempotence=true`, `transactional.id` set	`isolation.level=read_committed`, offsets committed inside the producer transaction

The producer’s acks setting controls durability: acks=all waits for all in-sync replicas. Idempotence (enable.idempotence=true, the default since Kafka 3.0) prevents retries from creating duplicate records on the broker. On the consumer side, the dividing line between at-most-once and at-least-once is purely when you commit relative to processing.

At-least-once consumer

Disable auto-commit and commit only after the work is durable.

var props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-service");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

try (var consumer = new KafkaConsumer<String, String>(props)) {
    consumer.subscribe(List.of("orders"));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            process(record);            // do the work first
        }
        consumer.commitSync();          // then commit — at-least-once
    }
}

In Spring Boot, set the container’s ack mode and let the framework commit after the listener returns successfully.

spring:
  kafka:
    consumer:
      enable-auto-commit: false
      isolation-level: read_committed
    listener:
      ack-mode: RECORD
    producer:
      acks: all
      properties:
        enable.idempotence: true

@Component
public class OrderListener {

    @KafkaListener(topics = "orders", groupId = "orders-service")
    public void onOrder(OrderEvent event) {
        // Throwing here prevents the offset commit, so the record is redelivered.
        repository.save(event);
    }
}

public record OrderEvent(String orderId, BigDecimal amount) {}

Verifying the guarantee

You can observe redelivery by killing a consumer between processing and commit. With acks=all and idempotence on, the producer side never drops or duplicates on the broker; the consumer side then determines the final guarantee.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group orders-service

Output:

GROUP          TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
orders-service orders  0          1042            1045            3

A non-zero LAG after a crash means those three records are uncommitted and will be reprocessed on restart — the signature of at-least-once.

Auto-commit (enable.auto.commit=true) commits on a timer regardless of whether processing finished. It quietly produces at-most-once on crash. If you care about not losing messages, turn it off.

Best practices

Treat the guarantee as end-to-end: the consumer’s commit timing can cancel out a perfectly durable producer.
Default to at-least-once (acks=all, idempotent producer, manual commit after processing) unless you have a concrete reason not to.
Make consumers idempotent with natural dedup keys or upserts so redelivery is harmless.
Keep enable.idempotence=true on producers — it is free duplicate protection and the default in modern Kafka.
Reach for exactly-once (transactions) only for true read-process-write pipelines where duplicates corrupt results.
Set consumers to isolation.level=read_committed when reading from transactional producers so aborted records are never seen.
Monitor consumer lag and rebalance frequency; surprising reprocessing usually traces back to commit timing or rebalances.