Idempotent Producer

Network retries are a fact of life in distributed systems, and they are also the classic source of duplicate records in Kafka. When a producer sends a batch, the broker may persist it successfully but fail to return the acknowledgement before a timeout fires, prompting the producer to resend the same data. The idempotent producer solves this at the protocol level: with enable.idempotence=true the broker can recognise and discard re-sent batches, giving you exactly-once delivery semantics for the produce step without any application-level deduplication.

What “idempotence” actually guarantees

An idempotent producer guarantees that each record is written to its target partition exactly once and in order, even across retries, for the lifetime of a single producer session. It does not magically deduplicate messages your application sends twice on purpose, and it does not span multiple partitions or topics atomically — that stronger guarantee requires transactions. Think of idempotence as “retries can no longer create duplicates,” which is precisely the bug it was designed to eliminate.

Since Kafka 3.0 (clients) idempotence is enabled by default. If you have never touched enable.idempotence, you are most likely already running an idempotent producer.

How it works: producer IDs and sequence numbers

When an idempotent producer initialises, the broker assigns it a unique Producer ID (PID). For every partition the producer writes to, it attaches a monotonically increasing sequence number to each batch, starting at zero. The broker tracks the highest sequence number it has committed per (PID, partition) pair.

The deduplication logic is simple and lives entirely on the broker:

Incoming batch seq == last committed seq + 1   -> accept, advance counter
Incoming batch seq <= last committed seq        -> duplicate, ACK but discard
Incoming batch seq >  last committed seq + 1     -> OutOfOrderSequenceException

So if a retry arrives carrying a sequence number the broker has already seen, the broker silently drops the payload but still returns a success acknowledgement — the producer is happy, and no duplicate lands in the log. The out-of-order case is what lets Kafka also preserve ordering: a gap means an earlier batch was lost, and the producer is told to resend rather than write records out of sequence.

Required configuration

Idempotence is not free-floating; it forces a set of consistent settings. If you enable it explicitly and contradict any of these, the producer throws a ConfigException at startup rather than silently degrading.

Property	Required value	Why
`enable.idempotence`	`true`	Turns the feature on (default in modern clients).
`acks`	`all` (`-1`)	The broker must confirm all in-sync replicas before sequence state is durable.
`max.in.flight.requests.per.connection`	`<= 5`	More than 5 unacked requests would break ordering guarantees.
`retries`	`> 0`	Idempotence exists to make retries safe; zero retries makes it pointless.

With idempotence on, these defaults are applied automatically, so the typical configuration is minimal.

# producer.properties — modern defaults shown explicitly for clarity
enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
retries=2147483647
delivery.timeout.ms=120000

The combination of high retries and a bounded delivery.timeout.ms is intentional: the producer retries aggressively but gives up on a record once the total time budget is exhausted, surfacing a TimeoutException to your callback instead of retrying forever.

Configuring it in a plain Java client

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class IdempotentProducerApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // Idempotence and its required companions
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

        try (Producer<String, String> producer = new KafkaProducer<>(props)) {
            ProducerRecord<String, String> record =
                    new ProducerRecord<>("orders", "order-42", "{\"total\":99.50}");
            RecordMetadata md = producer.send(record).get();
            System.out.printf("Sent to %s-%d @ offset %d%n",
                    md.topic(), md.partition(), md.offset());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Output:

Sent to orders-0 @ offset 17

Even if the underlying network call had to retry several times, exactly one record at offset 17 exists in the log.

Configuring it in Spring Boot

Spring for Apache Kafka exposes the same client properties under spring.kafka.producer. Idempotence is the default, but pinning it explicitly documents intent for production services.

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      acks: all
      retries: 2147483647
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5
        delivery.timeout.ms: 120000

A standard KafkaTemplate then sends idempotently with no extra code:

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

public record OrderEvent(String orderId, double total) { }

@Service
public class OrderPublisher {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public OrderPublisher(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publish(OrderEvent event) {
        kafkaTemplate.send("orders", event.orderId(), event);
    }
}

Limits to keep in mind

Idempotence is scoped to a single producer session. If the producer process restarts, it receives a new PID, so the broker can no longer correlate the new instance’s batches with the old ones. A record that was sent-but-unacknowledged before the crash could therefore be re-sent under the new PID and duplicated. To get end-to-end exactly-once across restarts you need a stable transactional.id and the transactional API, which ties idempotence to a durable identity.

Best Practices

Leave enable.idempotence=true (the default) on for virtually every producer — the overhead is negligible and the duplicate protection is real.
Never set acks=0 or acks=1 while expecting idempotence; the broker needs acks=all to keep its sequence state durable across replicas.
Keep max.in.flight.requests.per.connection at 5 or below; raising it disables ordering and idempotence simultaneously.
Prefer bounding delivery.timeout.ms over capping retries, so retries stop based on real wall-clock time rather than an arbitrary attempt count.
Treat idempotence as protection against retries, not against application-level resends — design consumers to be idempotent too for true end-to-end safety.
When you need atomic multi-partition writes or exactly-once across restarts, graduate from idempotence to transactions with a stable transactional.id.