At-Least-Once Processing

At-least-once is the delivery guarantee most production Kafka pipelines actually rely on: every record is processed at least one time, and never silently lost. The trade-off is that records may occasionally be processed more than once — typically after a crash or consumer-group rebalance. The recipe is simple to state but easy to get subtly wrong: durable writes on the producer (acks=all plus retries) and commit-after-processing on the consumer, backed by a consumer that treats redelivery as harmless. Get those two halves right and you have a pipeline that does not lose data under realistic failure conditions.

The two halves of the guarantee

At-least-once is an end-to-end property. It only holds if both the producer and the consumer cooperate.

On the producer side you must ensure a record is durably committed to enough replicas before you treat the send as successful. That means acks=all so the leader waits for all in-sync replicas, plus enough retries to ride out transient leader elections. Without retries, a transient NotEnoughReplicasException or leader change drops the message and you are silently below “at-least-once.”

On the consumer side you must commit the offset only after the side effects of processing are durable — after the database write, after the downstream publish. If you commit first and then crash, the record is lost (that is at-most-once). Commit last, and a crash simply means the record is redelivered: at-least-once.

Producer (acks=all, retries) --> Topic (replicated) --> Consumer
                                                          1. poll()
                                                          2. process + persist
                                                          3. commit offset   <-- commit LAST

Producer configuration

# Wait for all in-sync replicas to acknowledge
acks=all
# Retry transient failures (default in modern clients is high)
retries=2147483647
delivery.timeout.ms=120000
# Bound the retry window without dropping records
request.timeout.ms=30000
# Idempotent producer prevents duplicates from producer-side retries
enable.idempotence=true

Enabling enable.idempotence=true is the default in recent Kafka clients and is strongly recommended. It de-duplicates producer retries so a network hiccup during a retry does not write the record twice. It does not remove consumer-side duplicates from redelivery — that is what idempotent consumers are for.

On the broker, pair this with min.insync.replicas=2 on a topic with replication.factor=3 so acks=all has real meaning. See min.insync.replicas for the durability math.

Why duplicates still happen

Even with a perfect producer, the consumer side introduces duplicates whenever processing succeeds but the offset commit does not. Concretely:

The consumer processes records and writes to the database, then crashes before committing. On restart it re-polls from the last committed offset and reprocesses them.
A rebalance revokes partitions after processing but before the commit lands; the new owner reprocesses.
max.poll.interval.ms is exceeded because a batch took too long, the consumer is ejected, and the redelivered batch is processed by another member.

These are not bugs — they are the cost of not losing data. The correct response is not to chase zero duplicates (that requires exactly-once semantics) but to make duplicates harmless.

Designing an idempotent consumer

Idempotency means processing the same record twice produces the same result as processing it once. Two common strategies:

Strategy	How it works	Best for
Dedup key / processed-ID table	Store a unique message ID; skip records already seen	Side effects that can’t be made naturally idempotent (e.g. emails, external calls)
Upsert by natural key	Write with `INSERT ... ON CONFLICT DO UPDATE` keyed on a business key	Database state that converges to the same value

The cleanest pattern is an upsert keyed on a stable business identifier so reprocessing converges to the same row:

INSERT INTO orders (order_id, status, amount, updated_at)
VALUES (?, ?, ?, ?)
ON CONFLICT (order_id)
DO UPDATE SET status = EXCLUDED.status,
              amount = EXCLUDED.amount,
              updated_at = EXCLUDED.updated_at;

Spring Boot: commit-after-processing

Configure manual acknowledgment so the offset is committed only after your listener returns successfully. Spring for Apache Kafka handles redelivery and the rebalance lifecycle for you.

spring:
  kafka:
    consumer:
      enable-auto-commit: false
      auto-offset-reset: earliest
    listener:
      ack-mode: manual
    producer:
      acks: all
      properties:
        enable.idempotence: true

public record OrderEvent(String orderId, String status, long amountCents) {}

@Component
public class OrderConsumer {

    private final OrderRepository orders;

    public OrderConsumer(OrderRepository orders) {
        this.orders = orders;
    }

    @KafkaListener(topics = "orders", groupId = "order-processor")
    public void onMessage(OrderEvent event, Acknowledgment ack) {
        // Idempotent upsert: safe to run more than once for the same orderId
        orders.upsert(event.orderId(), event.status(), event.amountCents());

        // Commit ONLY after the side effect is durable.
        // A crash before this line simply redelivers the record.
        ack.acknowledge();
    }
}

If upsert throws, ack.acknowledge() is never reached, the offset stays put, and the record is redelivered — exactly the behaviour at-least-once promises. Because the write is an upsert, the redelivery is harmless.

Output:

Received OrderEvent[orderId=A-1001, status=PAID, amountCents=4999]
Upserted order A-1001 (1 row affected)
Committed offset 41 for orders-0

Dead-letter handling for poison records

A record that always fails will be redelivered forever, blocking the partition. Route persistent failures to a dead-letter topic after a bounded number of attempts using a DefaultErrorHandler.

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    // 3 retries with 1s backoff, then publish to <topic>.DLT
    return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3L));
}

Best Practices

Set acks=all, keep enable.idempotence=true, and run topics with replication.factor=3 and min.insync.replicas=2.
Always commit offsets after side effects are durable; use ack-mode: manual (or MANUAL_IMMEDIATE) rather than auto-commit.
Make every consumer idempotent — prefer upserts on a business key, or a dedup table keyed on a stable message ID.
Carry an idempotency key in the event payload or a header so reprocessing can be detected deterministically.
Keep per-record work below max.poll.interval.ms; offload slow operations to avoid eviction-driven duplicates.
Add a dead-letter topic with bounded retries so poison messages don’t stall a partition forever.
Don’t reach for transactions or exactly-once semantics unless idempotent at-least-once genuinely can’t model your side effects — it’s simpler, faster, and sufficient most of the time.