Topic Management

Topics are the fundamental unit of organization in Kafka, and getting their layout right is one of the highest-leverage decisions you make in production. Partition count drives parallelism and throughput, replication factor drives durability, and per-topic configs like retention and cleanup policy govern how data lives and dies. This page covers the day-to-day operations of creating, inspecting, altering, and deleting topics from both the command line and the programmatic AdminClient, with the operational caveats that trip teams up.

Creating topics from the CLI

The kafka-topics.sh script (kafka-topics.bat on Windows) is the workhorse for topic lifecycle operations. In KRaft mode you point it at a broker via --bootstrap-server; there is no ZooKeeper connection string anymore.

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic orders \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config cleanup.policy=delete

Output:

Created topic orders.

Choose the partition count carefully: it sets the maximum consumer parallelism for a consumer group, and while you can increase it later, you can never decrease it. The replication factor must be less than or equal to the number of brokers; 3 is the standard production value for tolerating one broker failure with one in-sync replica to spare.

Inspecting topics

Use --describe to see partitions, the leader, the replica assignment, and the in-sync replica (ISR) set. Run it without --topic to list every topic in the cluster.

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

Output:

Topic: orders   TopicId: 7Hk2... PartitionCount: 6   ReplicationFactor: 3   Configs: cleanup.policy=delete,retention.ms=604800000
        Topic: orders   Partition: 0    Leader: 1   Replicas: 1,2,3 Isr: 1,2,3
        Topic: orders   Partition: 1    Leader: 2   Replicas: 2,3,1 Isr: 2,3,1
        Topic: orders   Partition: 2    Leader: 3   Replicas: 3,1,2 Isr: 3,1,2
        ...

A healthy topic shows Isr equal to Replicas for every partition. A shrinking ISR is an early signal of an overloaded or failing broker.

Increasing partitions and the ordering caveat

You can grow a topic’s partition count with --alter, but you can only ever increase it.

kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic orders --partitions 12

Warning: Adding partitions changes the output of the default partitioner. Records are routed by hash(key) % partitionCount, so a key that landed on partition 3 with 6 partitions may map to a different partition with 12. This breaks per-key ordering for keys that were in flight. If you depend on key ordering, create the topic with enough partitions up front, or stop producers and drain consumers before resizing.

Managing per-topic configs

Most operational tuning happens through dynamic topic configs rather than recreating the topic. The kafka-configs.sh tool reads and writes these without restarting brokers.

# Show effective configs for a topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders --describe

# Switch a topic to log compaction and tighten retention
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter --add-config cleanup.policy=compact,retention.ms=86400000

# Remove an override and fall back to the broker default
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter --delete-config retention.ms

The most frequently changed topic configs:

Config	Default	Purpose
`retention.ms`	604800000 (7 days)	How long to keep records before deletion.
`retention.bytes`	-1 (unlimited)	Max size per partition before old segments are pruned.
`cleanup.policy`	`delete`	`delete` ages out data; `compact` keeps the latest value per key; `compact,delete` does both.
`min.insync.replicas`	1	ISR required for `acks=all` writes to succeed.
`max.message.bytes`	1048588	Largest single record accepted.
`segment.ms`	604800000	How often a new log segment is rolled.

Programmatic management with AdminClient

For automation, provisioning, and tests, the org.apache.kafka:kafka-clients AdminClient exposes the same operations with futures you can compose. Calls are asynchronous; call .all().get() to block on completion.

import org.apache.kafka.clients.admin.*;
import org.apache.kafka.common.config.ConfigResource;
import org.apache.kafka.common.config.ConfigResource.Type;

import java.util.List;
import java.util.Map;
import java.util.Properties;

public class TopicAdmin {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (Admin admin = Admin.create(props)) {
            // Create a topic with explicit configs
            NewTopic orders = new NewTopic("orders", 6, (short) 3)
                    .configs(Map.of(
                            "retention.ms", "604800000",
                            "cleanup.policy", "delete"));
            admin.createTopics(List.of(orders)).all().get();

            // Increase partitions to 12
            admin.createPartitions(
                    Map.of("orders", NewPartitions.increaseTo(12))).all().get();

            // Alter a config incrementally
            ConfigResource resource = new ConfigResource(Type.TOPIC, "orders");
            AlterConfigOp op = new AlterConfigOp(
                    new ConfigEntry("cleanup.policy", "compact"),
                    AlterConfigOp.OpType.SET);
            admin.incrementalAlterConfigs(
                    Map.of(resource, List.of(op))).all().get();

            // Delete the topic
            admin.deleteTopics(List.of("orders")).all().get();
        }
    }
}

In Spring Boot, declaring NewTopic beans lets the KafkaAdmin auto-create topics on startup, which is convenient for development and tests:

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.TopicBuilder;

@Configuration
public class KafkaTopicConfig {

    @Bean
    NewTopic ordersTopic() {
        return TopicBuilder.name("orders")
                .partitions(6)
                .replicas(3)
                .config("retention.ms", "604800000")
                .build();
    }
}

Deleting topics

Deletion is permanent and asynchronous; the controller marks the topic for deletion and reclaims its log segments in the background.

kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic orders

Tip: Deletion only works when the broker setting delete.topic.enable=true (the default in modern Kafka). In locked-down production clusters this is sometimes disabled to prevent accidental data loss, in which case --delete returns an error and you must re-enable it deliberately.

Best Practices

Provision partition counts for your peak parallelism up front; resizing later silently breaks per-key ordering.
Use replication factor 3 with min.insync.replicas=2 and acks=all for durable production topics.
Prefer dynamic config changes via kafka-configs.sh or incrementalAlterConfigs over deleting and recreating topics.
Pick cleanup.policy=compact for keyed state/changelog topics and delete for event streams; mix both with compact,delete when you need a compacted topic with a time bound.
Codify topic creation in version-controlled automation (AdminClient or IaC) rather than ad-hoc CLI runs, so environments stay reproducible.
Monitor the ISR set after any partition or replica change; a persistently shrunk ISR means writes with acks=all may be blocked or at risk.