Consumer Groups

Consumer groups are the mechanism Kafka uses to scale consumption horizontally while preserving ordering guarantees per partition. Every consumer that shares the same group.id cooperates to read a topic exactly once across the group, with the broker dividing partitions among members. Understanding this model is essential in production, because it dictates your maximum parallelism, how throughput scales when you add instances, and what happens when a pod dies or a deployment rolls.

What a consumer group is

A consumer group is a logical set of consumer instances identified by a single string, group.id. The group coordinator (a broker elected per group) tracks membership and assigns partitions so that each partition is consumed by exactly one member of the group at a time. This is the core invariant: within a group there is no double-processing of a partition, which is how Kafka delivers ordered, load-balanced consumption.

Different groups are independent. Two groups subscribed to the same topic each receive a full copy of every record — that is how you fan a stream out to multiple downstream applications (for example, a billing service and an analytics service) without them interfering.

# Members of the SAME group share this value and split the partitions.
group.id=order-processor
bootstrap.servers=broker1:9092,broker2:9092
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

Partitions are the unit of parallelism

The number of partitions on a topic is the hard ceiling on useful parallelism within a group. If a topic has 4 partitions, at most 4 consumers in the group do work; a 5th and 6th sit idle with no assignment, ready to take over only if an active member leaves.

Topic "orders" (4 partitions), group "order-processor" with 3 members:

   ┌───────────── orders ─────────────┐
   │  P0     P1     P2     P3          │
   └──┬──────┬──────┬──────┬───────────┘
      │      │      │      │
      ▼      ▼      ▼      ▼
   ┌──────┐ ┌──────┐ ┌─────────────┐
   │ C1   │ │ C2   │ │ C3          │
   │ P0   │ │ P1   │ │ P2, P3      │   <- C3 owns two partitions
   └──────┘ └──────┘ └─────────────┘

Add a 4th consumer -> each owns exactly one partition (perfect balance).
Add a 5th consumer -> it gets NO partitions and stays idle.

Provision partitions for your future peak, not today’s load. You can scale consumers up to the partition count instantly, but adding partitions later changes key-to-partition mapping and breaks per-key ordering for in-flight data.

Running multiple instances

You scale a group simply by starting more processes (or pods) that use the same group.id. No code change is required — the coordinator triggers a rebalance and redistributes partitions. Below is a minimal poll loop; running it twice with the same group splits the partitions between the two JVMs.

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

public class OrderConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "broker1:9092");
        props.put("group.id", "order-processor");
        props.put("key.deserializer",
            "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer",
            "org.apache.kafka.common.serialization.StringDeserializer");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));
            while (true) {
                ConsumerRecords<String, String> records =
                    consumer.poll(Duration.ofMillis(500));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("instance handled partition=%d offset=%d key=%s%n",
                        record.partition(), record.offset(), record.key());
                }
                consumer.commitSync();
            }
        }
    }
}

In Spring for Apache Kafka the same idea is expressed declaratively. Setting concurrency runs that many consumer threads inside one application, and each thread is a distinct group member.

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class OrderListener {

    // groupId ties this listener to the "order-processor" group.
    // concurrency = 3 starts three members from a single instance.
    @KafkaListener(topics = "orders", groupId = "order-processor", concurrency = "3")
    public void onMessage(String payload) {
        System.out.println("Processing: " + payload);
    }
}

You can confirm the live assignment with the CLI:

kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group order-processor

Output:

GROUP            TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID        HOST
order-processor  orders  0          1502            1502            0    consumer-1-af3...  /10.0.0.11
order-processor  orders  1          1488            1490            2    consumer-1-af3...  /10.0.0.11
order-processor  orders  2          1499            1499            0    consumer-2-9c1...  /10.0.0.12
order-processor  orders  3          1471            1480            9    consumer-3-72b...  /10.0.0.13

Group size versus partition count

Members vs. partitions	Result	Typical use
Members < partitions	Some members own multiple partitions	Normal steady state
Members == partitions	One partition per member, max parallelism	Peak throughput
Members > partitions	Extra members idle, no assignment	Hot standby / faster failover

Idle standbys are not always waste: when an active member crashes, a standby is assigned its partitions on the next rebalance, shortening recovery time.

Best practices

Pick a stable, descriptive group.id per logical application; never reuse one group id across unrelated services that must each see the full stream.
Size partition count for peak concurrency plus headroom, since members can never exceed partitions for useful work.
Scale by running more instances of the same group rather than changing code; let the coordinator rebalance.
Keep poll-loop processing fast (or use a larger max.poll.interval.ms) so members are not evicted mid-batch and trigger needless rebalances.
Monitor per-partition lag from kafka-consumer-groups.sh or JMX to know when to add members.
Use a few idle standbys when fast failover matters more than resource efficiency.
Give consumers a stable identity with static group membership to avoid rebalances on routine restarts.