Async Sends & Callbacks

Every KafkaProducer.send() call is asynchronous under the hood: the record is appended to an in-memory buffer and a background I/O thread batches it to the broker. The method hands you back a Future<RecordMetadata> so you can decide how to learn the outcome — fire-and-forget, register a callback, or block on the future. Choosing the right pattern is one of the most consequential decisions for a producer’s throughput and durability, so it pays to understand exactly what each style guarantees.

How send() actually works

send() returns immediately after the record is serialized, partitioned, and placed in the accumulator buffer. The actual network transmission happens later on the sender thread, governed by linger.ms and batch.size. The returned Future<RecordMetadata> is only completed once the broker acknowledges the record according to your acks setting.

ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", "order-42", "{\"total\":129.90}");

Future<RecordMetadata> future = producer.send(record);
// send() has returned, but the record may not be on the broker yet

Because the work is deferred, an exception thrown from send() itself only signals a synchronous failure (serialization error, buffer exhaustion after max.block.ms, illegal record). Delivery failures arrive later — through the future or a callback.

Fire-and-forget

The simplest pattern ignores the future entirely. It maximizes throughput but gives you no visibility into delivery failures beyond the producer’s internal retries.

producer.send(new ProducerRecord<>("metrics", "cpu", "0.73"));

This is acceptable only for data where occasional loss is tolerable (high-volume telemetry, non-critical logs). For anything you care about, prefer a callback.

Fire-and-forget does not mean “no retries.” The producer still retries retriable errors per retries/delivery.timeout.ms. What you lose is awareness of records that exhaust those retries and are dropped.

Asynchronous send with a callback

The recommended production pattern passes a Callback whose onCompletion(RecordMetadata, Exception) fires on the producer’s I/O thread once the record is acknowledged or has permanently failed. Exactly one of the two arguments is non-null.

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        log.error("Delivery failed for key {}: {}",
                record.key(), exception.getMessage(), exception);
    } else {
        log.info("Sent to {}-{} @offset {} (ts={})",
                metadata.topic(), metadata.partition(),
                metadata.offset(), metadata.timestamp());
    }
});

Output:

Sent to orders-2 @offset 10473 (ts=1717243200123)

Callbacks for records sent to the same partition execute in send order, which lets you reason about ordering when reacting to results. Because the callback runs on the shared I/O thread, keep it fast — never block, call send() recursively in a tight loop, or do heavy work inside it, or you will stall the entire producer.

Distinguishing retriable from fatal errors

In the callback you often want different handling for transient versus permanent failures. The Kafka client marks recoverable errors with the RetriableException marker interface.

producer.send(record, (metadata, exception) -> {
    if (exception == null) return;
    if (exception instanceof RetriableException) {
        log.warn("Retriable failure (client will retry): {}", exception.getMessage());
    } else {
        deadLetterService.park(record);   // route to DLQ, alert, etc.
        log.error("Fatal send error", exception);
    }
});

With delivery.timeout.ms and retries configured, the client already retries RetriableExceptions for you; the callback only surfaces them once retries are exhausted.

Synchronous send with get()

Calling get() on the returned future blocks the calling thread until the broker acknowledges, turning the async API into a synchronous one. This is the strongest delivery guarantee but the slowest — you serialize round trips one record at a time.

try {
    RecordMetadata md = producer.send(record).get();
    log.info("Confirmed offset {}", md.offset());
} catch (ExecutionException e) {
    // delivery failed — unwrap the real cause
    throw new KafkaPublishException("send failed", e.getCause());
} catch (InterruptedException e) {
    Thread.currentThread().interrupt();
    throw new KafkaPublishException("interrupted while sending", e);
}

Note that delivery exceptions are wrapped in ExecutionException; the real cause (e.g. TimeoutException, RecordTooLargeException) is available via getCause().

Choosing a pattern

Pattern	Throughput	Failure visibility	Ordering	When to use
Fire-and-forget	Highest	None (only internal retries)	Per-partition by send order	Lossy, high-volume telemetry
Async + callback	High	Yes, on I/O thread	Per-partition by send order	Default for most services
`send().get()`	Lowest	Yes, on caller thread	Strict, one-at-a-time	Critical writes, request/response flows

Avoid the trap of calling get() inside a loop over each record. You get synchronous latency with none of the batching benefit. If you need confirmations for a batch, send them all first (collecting the futures), then drain the futures afterward.

List<Future<RecordMetadata>> futures = new ArrayList<>();
for (Order o : orders) {
    futures.add(producer.send(new ProducerRecord<>("orders", o.id(), o.json())));
}
for (Future<RecordMetadata> f : futures) {
    f.get();   // batches still flow concurrently; we just wait at the end
}

Best Practices

Use the callback style as your default — it gives delivery visibility without blocking the producer.
Keep callback bodies non-blocking and cheap; offload heavy work to another executor.
Branch on RetriableException in callbacks and route exhausted/fatal records to a dead-letter destination.
Reserve send().get() for truly critical, low-volume writes; never call it per-record inside a hot loop.
When you do need synchronous confirmation for many records, send first and join the futures afterward to preserve batching.
Always unwrap ExecutionException.getCause() to log and handle the real delivery error.
Remember a clean send() return is not delivery confirmation — only the future/callback tells you the record reached the broker.