Dead Letter Topics (DLT)

Even with retries and backoff, some records can never be processed successfully — a malformed payload, a permanently rejected downstream call, or a poison message that would otherwise block the partition forever. A dead letter topic (DLT) is the production-grade escape hatch: after a consumer exhausts its retry budget, the failing record is published to a side topic so the main flow keeps moving while the failure is preserved for inspection, alerting, or replay. Spring for Apache Kafka ships DeadLetterPublishingRecoverer to do exactly this, and wiring it into your DefaultErrorHandler takes only a few lines.

How the recoverer fits the error-handling pipeline

When a @KafkaListener throws, the container hands the record to its CommonErrorHandler. The default, DefaultErrorHandler, retries the record according to its BackOff. Once retries are exhausted, it invokes a recoverer — a BiConsumer<ConsumerRecord<?, ?>, Exception>. By default the recoverer just logs and moves on. By supplying a DeadLetterPublishingRecoverer, you instead republish the dead record to a dead-letter topic, then commit the offset so the consumer advances.

The recoverer needs a KafkaTemplate to publish with, so it should use the same serializers (or a byte-passthrough template) appropriate for your value type.

Wiring DeadLetterPublishingRecoverer into DefaultErrorHandler

Expose the recoverer and error handler as beans. The error handler is picked up automatically by the container factory when it’s a bean named appropriately, or you can set it explicitly.

package com.devcraftly.kafka.config;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.FixedBackOff;

@Configuration
public class KafkaErrorConfig {

    @Bean
    public DeadLetterPublishingRecoverer deadLetterRecoverer(
            KafkaTemplate<Object, Object> template) {
        // Default destination resolver: <topic>-<partition> on a topic named "<orig>.DLT"
        return new DeadLetterPublishingRecoverer(template);
    }

    @Bean
    public DefaultErrorHandler errorHandler(DeadLetterPublishingRecoverer recoverer) {
        // Retry 3 times, 1s apart, then publish to the DLT.
        DefaultErrorHandler handler =
                new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3L));

        // Don't retry unrecoverable errors — go straight to the DLT.
        handler.addNotRetryableExceptions(IllegalArgumentException.class);
        return handler;
    }
}

Make DefaultErrorHandler a bean: Spring Boot’s auto-configuration detects a single CommonErrorHandler bean and applies it to every ConcurrentKafkaListenerContainerFactory for you. If you build the factory manually, call factory.setCommonErrorHandler(errorHandler).

Default DLT naming

By default the recoverer routes a failed record from topic orders and partition 2 to the topic orders.DLT, partition 2 — it preserves the original partition number. If the DLT has fewer partitions than the source topic, that partition may not exist; in that case send to partition 0 by passing a custom destination resolver:

return new DeadLetterPublishingRecoverer(template,
        (record, ex) -> new org.apache.kafka.common.TopicPartition(
                record.topic() + ".DLT", 0));

The .DLT suffix is the convention used across Spring Kafka (including @RetryableTopic), so dead-letter consumers and tooling can discover these topics predictably.

Source topic	Source partition	Default DLT topic	Default DLT partition
`orders`	0	`orders.DLT`	0
`orders`	2	`orders.DLT`	2
`payments`	5	`payments.DLT`	5

Headers added to the dead record

DeadLetterPublishingRecoverer enriches the republished record with diagnostic headers so you can reconstruct exactly what failed without digging through logs. The keys are defined in KafkaHeaders:

Header	Meaning
`kafka_dlt-exception-fqcn`	Fully-qualified exception class name
`kafka_dlt-exception-message`	Exception message
`kafka_dlt-exception-stacktrace`	Full stack trace
`kafka_dlt-original-topic`	Original topic the record came from
`kafka_dlt-original-partition`	Original partition
`kafka_dlt-original-offset`	Original offset
`kafka_dlt-original-timestamp`	Original record timestamp

The original key and value bytes are carried through unchanged, so the DLT record is a faithful copy plus failure metadata.

Consuming the DLT to inspect failures

Attach a @KafkaListener to the dead-letter topic to log, alert, or persist failures. Read the diagnostic headers directly.

package com.devcraftly.kafka.listener;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.messaging.handler.annotation.Payload;
import org.springframework.stereotype.Component;

@Component
public class OrdersDltListener {

    private static final Logger log = LoggerFactory.getLogger(OrdersDltListener.class);

    @KafkaListener(topics = "orders.DLT", groupId = "orders-dlt-monitor")
    public void onDeadLetter(
            @Payload(required = false) String payload,
            @Header(KafkaHeaders.DLT_ORIGINAL_TOPIC) String origTopic,
            @Header(KafkaHeaders.DLT_ORIGINAL_PARTITION) int origPartition,
            @Header(KafkaHeaders.DLT_ORIGINAL_OFFSET) long origOffset,
            @Header(KafkaHeaders.DLT_EXCEPTION_FQCN) String exceptionClass,
            @Header(KafkaHeaders.DLT_EXCEPTION_MESSAGE) String exceptionMessage) {

        log.error("Dead letter from {}-{}@{}: {} -> {} | payload={}",
                origTopic, origPartition, origOffset,
                exceptionClass, exceptionMessage, payload);
        // Persist to a failures table, raise an alert, or queue for replay.
    }
}

Output:

ERROR c.d.k.listener.OrdersDltListener - Dead letter from orders-2@1487:
  java.lang.IllegalArgumentException -> price must be positive |
  payload={"orderId":"A-91","price":-5}

The .DLT topic is just another Kafka topic — auto-creation must be enabled on the broker, or create it explicitly (KafkaAdmin / NewTopic) with the same partition count as the source to keep partition mapping intact.

Best Practices

Make DefaultErrorHandler and DeadLetterPublishingRecoverer Spring beans so auto-configuration applies the handler to every listener consistently.
Pre-create each .DLT topic with the same partition count as its source topic so the default partition-preserving resolver always has a valid target.
Use addNotRetryableExceptions(...) for deserialization and validation errors — retrying a poison message wastes the entire backoff budget before it inevitably hits the DLT.
Monitor DLT lag and depth in your observability stack; a growing DLT is a leading indicator of a systemic bug or a bad deploy.
Keep a deliberate replay path (re-publish from .DLT back to the source topic after a fix) rather than consuming and silently discarding failures.
Set a generous but finite retention.ms on DLT topics so failures survive long enough to investigate without growing unbounded.