Producer Tuning

The Kafka producer is where most end-to-end performance is won or lost, because it decides how records are batched, compressed, and acknowledged before they ever reach a broker. A handful of settings — batch.size, linger.ms, compression.type, buffer.memory, max.in.flight.requests.per.connection, acks, delivery.timeout.ms, and idempotence — control the trade-off between throughput, latency, and durability. The right values are not universal; they depend on whether you are optimizing for raw ingest rate or for the lowest possible per-message delay. This page is a checklist for setting them deliberately rather than accepting the defaults.

How the producer batches and sends

When you call send(), the record is serialized and appended to an in-memory batch keyed by topic-partition. The producer holds that batch until one of two things happens: it fills to batch.size, or linger.ms elapses since the batch was created. Only then is the batch handed to the I/O thread, optionally compressed, and shipped to the broker. All batches share a single buffer.memory pool; when that pool is exhausted, send() blocks (up to max.block.ms) waiting for space.

send() -> serialize -> append to partition batch
                          |
        full (batch.size) OR aged (linger.ms)
                          v
        compress (compression.type) -> network -> broker

Understanding this flow makes the tuning levers obvious: bigger, longer-lived batches mean higher throughput but more latency; smaller, eagerly-flushed batches mean lower latency but more overhead per record.

The core settings

Property	Default	What it controls	Tuning direction
`batch.size`	16384 (16 KB)	Max bytes per partition batch	Up for throughput, down/0 for latency
`linger.ms`	0	How long to wait for a batch to fill	Up for throughput, 0 for latency
`compression.type`	none	On-wire/on-disk compression codec	`zstd`/`lz4` for throughput, `none`/`lz4` for latency
`buffer.memory`	33554432 (32 MB)	Total producer buffer pool	Up under bursty or slow-broker conditions
`max.in.flight.requests.per.connection`	5	Unacked requests per connection	5 with idempotence keeps ordering
`acks`	all	Broker acknowledgements required	`all` for durability, `1` for lower latency
`delivery.timeout.ms`	120000	Total time a `send()` may take incl. retries	Bound to your SLA
`enable.idempotence`	true	Dedupes retries, preserves order	Keep `true`

Tip: Since Kafka 3.0, enable.idempotence defaults to true, which forces acks=all, retries > 0, and max.in.flight ≤ 5. Setting acks=1 while idempotence is enabled will throw a ConfigException — disable idempotence explicitly if you truly want weaker guarantees.

Two profiles, side by side

Most real systems land near one of two profiles. The table contrasts a high-throughput batch-ingest producer with a low-latency interactive producer.

Setting	High throughput	Low latency
`batch.size`	262144 (256 KB)	16384 (16 KB)
`linger.ms`	50–100	0
`compression.type`	`zstd`	`lz4` or `none`
`buffer.memory`	134217728 (128 MB)	33554432 (32 MB)
`acks`	`all`	`all` (or `1` if SLA demands)
`max.in.flight.requests.per.connection`	5	5
`enable.idempotence`	`true`	`true`
`delivery.timeout.ms`	120000	5000–10000

The high-throughput profile amortizes fixed per-request overhead across large, compressed batches. The low-latency profile sends immediately, accepting more requests-per-second and CPU in exchange for minimal queueing delay.

High-throughput producer (plain client)

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

props.put(ProducerConfig.BATCH_SIZE_CONFIG, 256 * 1024);            // 256 KB
props.put(ProducerConfig.LINGER_MS_CONFIG, 50);                     // let batches fill
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 128L * 1024 * 1024); // 128 MB
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    for (int i = 0; i < 1_000_000; i++) {
        producer.send(new ProducerRecord<>("events", Integer.toString(i), payload(i)));
    }
    producer.flush();
}

Low-latency producer (plain client)

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

props.put(ProducerConfig.LINGER_MS_CONFIG, 0);                      // send immediately
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16 * 1024);            // small batches
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");          // cheap CPU cost
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 5000);        // fail fast

The same profiles in Spring Boot

Spring for Apache Kafka exposes the well-known keys directly and passes anything else through properties.

spring:
  kafka:
    bootstrap-servers: broker1:9092,broker2:9092
    producer:
      acks: all
      batch-size: 262144         # 256 KB (high throughput)
      buffer-memory: 134217728   # 128 MB
      compression-type: zstd
      properties:
        linger.ms: 50
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5
        delivery.timeout.ms: 120000

For a typed event you can publish a Java record and let the JSON serializer handle it:

public record OrderPlaced(String orderId, long amountCents, Instant placedAt) {}

@Service
public class OrderPublisher {

    private final KafkaTemplate<String, OrderPlaced> kafkaTemplate;

    public OrderPublisher(KafkaTemplate<String, OrderPlaced> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publish(OrderPlaced event) {
        kafkaTemplate.send("orders", event.orderId(), event);
    }
}

Verifying your settings

After tuning, confirm the producer is actually batching and compressing by checking its JMX metrics. The console producer with --producer-property is a quick way to sanity-check codecs and acks end to end.

kafka-console-producer.sh --bootstrap-server broker1:9092 \
  --topic events \
  --producer-property compression.type=zstd \
  --producer-property acks=all

Then inspect throughput and batch metrics with kafka-producer-perf-test.sh:

kafka-producer-perf-test.sh --topic events --num-records 1000000 \
  --record-size 512 --throughput -1 \
  --producer-props bootstrap.servers=broker1:9092 \
  batch.size=262144 linger.ms=50 compression.type=zstd acks=all

Output:

1000000 records sent, 384615.4 records/sec (188.0 MB/sec), 42.1 ms avg latency, 312.0 ms max latency

A high batch-size-avg and a record-queue-time-avg close to your linger.ms confirm batches are filling as intended; a near-zero average batch size means records are flushing before they accumulate.

Best Practices

Start from one of the two profiles, then adjust a single setting at a time and re-measure — never tune batch.size, linger.ms, and compression.type blindly together.
Keep enable.idempotence=true (the default); it gives exactly-once-per-partition delivery and ordering at negligible cost, and lets you keep acks=all.
Use linger.ms as your throughput-vs-latency dial: a few milliseconds dramatically improves batching with barely noticeable added delay.
Pick zstd for the best compression ratio on text/JSON, lz4 when producer CPU is the bottleneck, and benchmark with real payloads — already-compressed binary will not shrink.
Size buffer.memory to absorb broker slowdowns so send() keeps batching instead of blocking; watch buffer-available-bytes for exhaustion.
Set delivery.timeout.ms to match your application SLA so failed sends surface promptly instead of retrying silently for two minutes.
Always call flush() (or close the producer) before shutdown so buffered batches are not lost on exit.