Capacity & Retention Planning

Kafka stores every message on disk for the duration of its retention policy, which makes storage — not CPU or memory — the resource most likely to take a cluster down. A broker whose log directory fills up will fail writes, drop out of the ISR, and can cascade into under-replicated partitions across the cluster. Capacity planning is the discipline of estimating how much disk you will consume, choosing retention that balances cost against data you actually need, and alerting long before the disk is full.

The core storage formula

For any topic, the steady-state disk footprint across the whole cluster is the product of how fast data arrives, how long you keep it, how many copies you store, and a fudge factor for indexes and unflushed segments:

disk = throughput_bytes_per_sec x retention_seconds x replication_factor x (1 + overhead)

The overhead term (typically 0.1 to 0.2) covers per-segment .index and .timeindex files, the active segment that has not yet rolled, and headroom so compaction and segment deletion have room to operate. Divide the cluster total by the number of brokers to get per-broker disk, assuming partitions are balanced.

Always compute storage in post-compression bytes. If producers use compression.type=lz4 or zstd, measure the actual on-disk size with kafka-log-dirs.sh rather than the raw event size — the difference is often 3-5x.

A worked example

Suppose an orders topic ingests 12,000 messages/sec at an average compressed size of 800 bytes, with replication.factor=3, and the business requires 7 days of retention.

throughput  = 12,000 msg/s x 800 B = 9.6 MB/s
retention   = 7 days = 604,800 s
raw         = 9.6 MB/s x 604,800 s = 5.81 TB
replicated  = 5.81 TB x 3          = 17.42 TB
with 15% OH = 17.42 TB x 1.15      = 20.0 TB

Spread across 6 brokers that is ~3.3 TB per broker for this single topic. Sum the figure for every topic, then size each broker’s disk so this total sits below your headroom threshold (see below).

Per-topic retention controls

Retention is configured per topic and overrides the broker defaults, so size-sensitive and compliance-sensitive topics can coexist on the same cluster. Time- and size-based limits work together: a segment is eligible for deletion when it exceeds retention.ms or the partition exceeds retention.bytes, whichever triggers first.

Config	Scope	Meaning	Typical value
`retention.ms`	per partition	Delete segments older than this	`604800000` (7 d)
`retention.bytes`	per partition	Cap total bytes per partition	`-1` (unlimited) or a hard cap
`segment.bytes`	per partition	Roll a new segment at this size	`1073741824` (1 GB)
`segment.ms`	per partition	Roll a new segment after this time	`604800000`
`cleanup.policy`	per topic	`delete`, `compact`, or `compact,delete`	`delete`

Setting retention.bytes is your safety net: it guarantees a partition can never exceed a known size even if throughput spikes far above your estimate. Set it slightly above your expected steady state so normal traffic uses time-based deletion while a runaway producer is bounded.

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders --alter \
  --add-config retention.ms=604800000,retention.bytes=536870912000,segment.bytes=1073741824

Compaction for changelog topics

Topics that represent the current state of a key — user profiles, account balances, Kafka Streams changelog/state-store topics — should use log compaction instead of (or alongside) time deletion. Compaction keeps only the latest value per key, so storage grows with the number of distinct keys rather than the number of writes.

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name user-profiles --alter \
  --add-config cleanup.policy=compact,min.cleanable.dirty.ratio=0.5,delete.retention.ms=86400000

Use cleanup.policy=compact,delete when you want compaction and a hard time bound (e.g. keep latest-per-key but never older than 30 days). For pure event-log topics, leave it as delete.

Planning growth headroom

Never plan for 100% disk usage. Compaction, recovery after a broker restart, and partition reassignment all need temporary spare space. A practical target is to keep steady-state usage at or below 60-70% of capacity, leaving room for traffic growth and operational events.

# Sizing target per broker
usable_disk: 8 TB
target_utilization: 0.65        # alert path begins above this
steady_state_per_broker: 5.2 TB # 8 TB x 0.65
growth_buffer: 6 months          # re-evaluate before usage trends past target

Track the trend, not just the instantaneous value. A disk at 50% that climbs 5% per week will hit the wall in three months — that is the signal to add brokers or trim retention, well ahead of an outage.

Alert before disks fill

Wire alerts to the JMX log-size metrics and the OS filesystem so you get paged with days of runway, not minutes. The key broker metric is kafka.log:type=LogManager log size plus the host node_filesystem_avail_bytes.

groups:
  - name: kafka-capacity
    rules:
      - alert: KafkaBrokerDiskWarning
        expr: (1 - node_filesystem_avail_bytes{mountpoint="/var/lib/kafka"}
               / node_filesystem_size_bytes{mountpoint="/var/lib/kafka"}) > 0.70
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: "Kafka data disk above 70% on {{ $labels.instance }}"
      - alert: KafkaBrokerDiskCritical
        expr: (1 - node_filesystem_avail_bytes{mountpoint="/var/lib/kafka"}
               / node_filesystem_size_bytes{mountpoint="/var/lib/kafka"}) > 0.85
        for: 5m
        labels: { severity: critical }

You can also use kafka-log-dirs.sh in a scheduled job to attribute usage to specific topics when an alert fires:

kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --topic-list orders,user-profiles

Output:

Querying brokers for log directories information
Received log directory information from brokers 1,2,3
{"brokers":[{"broker":1,"logDirs":[{"logDir":"/var/lib/kafka",
  "partitions":[{"partition":"orders-0","size":58210304512,"offsetLag":0}]}]}]}

Best Practices

Compute storage in compressed bytes using real on-disk measurements, not raw event sizes.
Set retention.bytes as a hard per-partition cap on every high-volume topic so a producer spike can never fill the disk.
Use cleanup.policy=compact for changelog/state topics and delete for event logs; combine them only when you need both bounds.
Keep steady-state disk usage at or below 60-70% to leave room for compaction, recovery, and reassignment.
Alert on both the utilization percentage and the rate of growth so you get days of warning before a disk fills.
Re-run the capacity formula whenever you add a topic or change retention, and review trends at least quarterly.
Right-size segment.bytes/segment.ms so segments roll often enough for timely deletion without creating millions of tiny files.