Offset Management

An offset is the monotonically increasing integer that identifies a record’s position inside a partition, and offset management is how a consumer remembers what it has already processed. Getting this right is the difference between a clean restart and either reprocessing thousands of records or silently skipping them. In production this single concept underpins your delivery semantics — at-least-once versus at-most-once — so it pays to understand exactly what Kafka stores, where, and when.

Position versus committed offset

A consumer tracks two distinct offsets per partition, and conflating them is the source of most “my messages vanished” incidents.

Position (the current position) is the offset of the next record the consumer will fetch. It lives only in the consumer’s memory and advances every time poll() returns records. It is volatile — when the process dies, it is gone.
Committed offset is the position the consumer has durably saved back to Kafka, marking “everything before this is done.” It survives restarts and is what a new owner of the partition reads to know where to resume.

These two values are usually different. After processing a batch you may have a position of 1500 (you’ve read up to 1499) but only committed 1450 because your last commit ran a few batches ago. On a crash you resume from 1450, so records 1450–1499 are processed again — that is at-least-once delivery.

Partition "orders-0" log:

  offset:  ... 1447 1448 1449 1450 1451 ... 1498 1499 | 1500 (next produce)
                              ^                     ^      ^
                              |                     |      |
                       committed offset       current position  log end offset
                         (saved 1450)         (in-memory 1500)   (high watermark)

  On restart, a new consumer reads the committed offset (1450)
  and re-fetches 1450..1499  ->  at-least-once.

You can inspect both at runtime with the Java client:

import org.apache.kafka.common.TopicPartition;

TopicPartition tp = new TopicPartition("orders", 0);
long position  = consumer.position(tp);                 // next record to fetch (in memory)
var committed  = consumer.committed(java.util.Set.of(tp)).get(tp); // last saved offset
System.out.printf("position=%d committed=%s%n",
    position, committed == null ? "none" : committed.offset());

Where offsets are stored: __consumer_offsets

Modern Kafka stores committed offsets in an internal compacted topic named __consumer_offsets. When a consumer commits, it produces a record to this topic keyed by (group.id, topic, partition); the value is the offset and optional metadata. Because the topic is log-compacted, only the latest committed offset per key is retained, so it never grows unbounded even after billions of commits.

This is a deliberate design choice: offsets are just another Kafka topic, replicated across brokers (offsets.topic.replication.factor, default 3) and read by the group coordinator. There is no separate database, and in KRaft mode no ZooKeeper is involved at all.

You can read the committed offsets for a group with the CLI:

kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group order-processor

Output:

GROUP            TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
order-processor  orders  0          1450            1500            50
order-processor  orders  1          2310            2310            0

Here CURRENT-OFFSET is the committed offset, LOG-END-OFFSET is the next offset the broker will assign, and LAG is the gap — work the group still has to do.

Never delete or hand-edit __consumer_offsets. If you need to move a group’s position, use kafka-consumer-groups.sh --reset-offsets (with --dry-run first); editing the topic directly corrupts every group on the cluster.

What happens on restart and rebalance

When a consumer instance restarts, or a rebalance reassigns a partition to a different member, the new owner asks the group coordinator for the committed offset of each partition it now owns and seeks there. Processing resumes from exactly that point — nothing in the consumer’s old in-memory position matters anymore.

This is why commit cadence equals your reprocessing window. If you commit after every batch, a crash replays at most one batch. If you commit once a minute, you may replay a minute of records. The trade-off is throughput: committing is a network round-trip, so very frequent synchronous commits cost latency.

A safe pattern is to commit after the work is durably done, so a crash mid-processing leaves the offset un-advanced and the records are retried:

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
    process(record);            // do the real work first
}
consumer.commitSync();          // only then mark progress -> at-least-once

When there is no committed offset

A group reads a partition for the first time — a brand-new group.id, or a partition whose committed offset has been deleted (e.g. by retention on offsets.retention.minutes, default 7 days of inactivity) — when there is nothing to resume from. Kafka resolves this with auto.offset.reset:

`auto.offset.reset`	Behavior when no committed offset exists
`latest` (default)	Start at the end; only consume records produced after the consumer joins
`earliest`	Start at the oldest retained record; replay the whole partition
`none`	Throw `NoOffsetForPartitionException` and fail fast

This setting only applies when no valid committed offset is found — once a group has committed, auto.offset.reset is irrelevant on subsequent restarts.

group.id=order-processor
enable.auto.commit=false
auto.offset.reset=earliest

A classic production bug: deploy a new consumer group with auto.offset.reset=latest, and it silently skips every record that existed before it started. Use earliest for groups that must process the full history, and none when starting from an unknown position should be treated as an error.

Best Practices

Treat position (in-memory) and committed (durable) as separate values; only the committed offset survives a restart or rebalance.
Commit after processing, not before, so a crash retries rather than skips — this is the foundation of at-least-once delivery.
Tune commit frequency to your acceptable reprocessing window; balance the replay cost against the latency of each commit round-trip.
Set auto.offset.reset=earliest for groups that must see full history, latest for live-only consumers, and none where an unknown start is a fatal error.
Never modify __consumer_offsets directly; use kafka-consumer-groups.sh --reset-offsets --dry-run to plan any reposition.
Watch offsets.retention.minutes for groups that idle for days — an expired offset silently falls back to auto.offset.reset.
Monitor lag (LOG-END-OFFSET - CURRENT-OFFSET) per partition to confirm offsets are advancing and the group is keeping up.