Partitions in Depth

Partitions are the unit of parallelism, ordering, and scale in Kafka — almost every performance and correctness decision you make traces back to how you partition. Pick too few and you cap throughput and consumer concurrency; pick too many and you strain broker memory, lengthen failover, and slow down metadata operations. Worse, the partition count is effectively a one-way door for keyed topics, because changing it reshuffles every key. This page covers how to size partitions, the real limits, the hot-partition problem, and why you should treat the partition count as a long-lived design decision.

Why partition count matters

A partition is a single ordered, append-only log. Within one partition Kafka guarantees order; across partitions there is no global ordering. Three things scale with partition count:

Producer throughput — writes to different partitions proceed in parallel across brokers and disks.
Consumer parallelism — within a consumer group, a partition is consumed by exactly one member. You can never have more active consumers than partitions; extra consumers sit idle.
Ordering granularity — records sharing a key land on the same partition, so per-key order is preserved only as long as the partition count is stable.

That last point is the trap. Kafka’s default partitioner computes partition = hash(key) % numPartitions. Change numPartitions and the modulo result changes for most keys, so a key that used to map to partition 3 now maps to partition 7 — its history is stranded behind it.

A sizing formula

Start from your target throughput and your per-partition ceiling. A single partition realistically sustains tens of MB/s (commonly ~10 MB/s for safe planning, more on fast NVMe). The classic LinkedIn rule of thumb:

partitions = max( target_throughput / producer_throughput_per_partition,
                  target_throughput / consumer_throughput_per_partition )

Worked example: you need 100 MB/s of sustained throughput. A producer pushes ~20 MB/s to a partition; a consumer drains ~10 MB/s after deserialization and processing.

producer side: 100 / 20 = 5 partitions
consumer side: 100 / 10 = 10 partitions
choose max     -> 10 partitions

Round up and add headroom for growth, since shrinking is impossible and growing is costly. For most topics, 6–12 partitions is a sane starting point; reserve high counts (50+) for genuinely high-volume streams.

Upper limits

There is no single hard cap, but partitions are not free. Each replica is an open file set and consumes broker memory and controller bookkeeping. Practical guidance for a healthy cluster:

Scope	Practical guideline	Why it bounds you
Per broker	up to ~1,000–4,000 partitions (replicas)	open files, memory, replication fetch threads
Per cluster (KRaft)	comfortably into the millions	KRaft metadata scales far better than ZooKeeper
Per cluster (ZooKeeper, legacy)	tens of thousands	controller failover replays all partition metadata
Failover cost	rises with leader count per broker	more leaders to re-elect when a broker dies

KRaft mode dramatically raised the ceiling versus ZooKeeper, but per-broker file-handle and memory limits still apply. Sizing for the cluster total is not the same as sizing for any single overloaded broker.

The hot partition problem

A hot partition gets a disproportionate share of traffic, usually from skewed keys — a “celebrity” user id, a single high-volume tenant, or a null/constant key. The owning broker and the one consumer assigned to it saturate while the rest of the cluster is idle. More partitions do not help if the skew is in the keys.

Mitigations, in order of preference:

Compound or salted keys — append a bounded suffix (userId-#{0..N}) to spread one logical key across several partitions, trading strict per-key order for throughput.
Custom partitioner — route deliberately instead of relying on the hash.
Null key when order is not required — Kafka then load-balances via the sticky partitioner.

public class TenantPartitioner implements Partitioner {

    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        int total = cluster.partitionCountForTopic(topic);
        String tenant = (String) key;
        // Give the noisy "global" tenant a 4-partition shard range;
        // everyone else hashes normally.
        if ("global".equals(tenant)) {
            return ThreadLocalRandom.current().nextInt(4);
        }
        return Math.floorMod(tenant.hashCode(), total - 4) + 4;
    }

    @Override public void close() { }
    @Override public void configure(Map<String, ?> configs) { }
}

Wire it into a Spring Boot producer factory via properties:

spring:
  kafka:
    producer:
      properties:
        partitioner.class: com.example.kafka.TenantPartitioner

The cost of adding partitions later

You can only ever increase partition count, never decrease it:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic orders --partitions 12

Output:

WARNING: If partitions are increased for a topic that has a key,
the partition logic or ordering of the messages will be affected

That warning is the whole story. After the change, hash(key) % 12 no longer matches the old hash(key) % 6. Existing records stay where they were, but new records for the same key may go elsewhere, so a consumer reading a single partition no longer sees that key’s full, ordered history. For event-sourced or order-sensitive topics this silently corrupts your guarantees.

Because of this, the safe pattern is to over-provision at creation time rather than scale up later. If you genuinely must re-partition a keyed topic, treat it as a migration: create a new topic with the target count and re-key/replay the data into it.

Best practices

Size from target throughput using the producer/consumer formula, then round up for growth headroom.
Keep at least as many partitions as your expected peak number of group consumers — idle consumers are wasted capacity.
Prefer a moderate count (6–12) for most topics; justify any topic above ~50 partitions with real throughput numbers.
Watch per-broker partition counts and partition-skew metrics; rebalance leaders before a broker becomes a hotspot.
Treat partition count on keyed topics as immutable — design for the future rather than planning to --alter later.
Solve hot partitions at the key level (salting, custom partitioner), not by blindly adding partitions.