Kafka Transport

Apache Kafka is a distributed, partitioned, replicated commit log built for high-throughput event streaming. NestJS ships a first-class Kafka transporter that lets a service consume from topics, produce records, and even model request-response on top of Kafka’s append-only log. Unlike point-to-point transports such as TCP or Redis, Kafka retains messages durably, so consumers can replay history and scale horizontally through consumer groups. This page shows how to configure the transporter, handle messages and events, implement the reply-topic pattern, and reason about partitioning.

Configuring the Kafka transporter

Under the hood NestJS uses KafkaJS, so the transporter options map directly to KafkaJS client, consumer, and producer configs. Attach the microservice in main.ts and give the client a stable clientId plus the consumer a groupId.

// main.ts
import { NestFactory } from '@nestjs/core';
import { MicroserviceOptions, Transport } from '@nestjs/microservices';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(
    AppModule,
    {
      transport: Transport.KAFKA,
      options: {
        client: {
          clientId: 'billing-service',
          brokers: ['kafka-1:9092', 'kafka-2:9092'],
        },
        consumer: {
          groupId: 'billing-consumer',
        },
      },
    },
  );

  await app.listen();
}
bootstrap();

The groupId is the most important setting. All instances sharing a groupId form one consumer group, and Kafka distributes the topic’s partitions across them — that is how you scale consumers and guarantee each record is processed once per group.

Consuming messages and events

@MessagePattern subscribes to a topic and is used for request-response (the handler’s return value is published to a reply topic). @EventPattern subscribes to a topic for fire-and-forget consumption and returns nothing. Use @Payload for the deserialized value and @Ctx to access the raw Kafka message, partition, and offset.

// orders.controller.ts
import { Controller } from '@nestjs/common';
import {
  Ctx,
  EventPattern,
  KafkaContext,
  MessagePattern,
  Payload,
} from '@nestjs/microservices';

@Controller()
export class OrdersController {
  @MessagePattern('order.created')
  handleOrderCreated(@Payload() order: { id: string; total: number }) {
    // The return value is sent to the 'order.created.reply' topic.
    return { id: order.id, status: 'INVOICED', total: order.total };
  }

  @EventPattern('order.shipped')
  handleOrderShipped(
    @Payload() event: { id: string },
    @Ctx() context: KafkaContext,
  ) {
    const message = context.getMessage();
    const topic = context.getTopic();
    const partition = context.getPartition();
    console.log(
      `[${topic}/${partition}@${message.offset}] shipped ${event.id}`,
    );
  }
}

Output:

[order.shipped/2@10487] shipped 0c3f9a12

By default the Kafka transporter only acknowledges offsets after the handler resolves successfully. Throwing inside an @EventPattern handler prevents the offset commit, so the record is redelivered on the next poll — design event handlers to be idempotent.

Producing records with ClientKafka

To emit events or send messages from another service, inject a ClientKafka provider. Register it with ClientsModule, then emit() for events and send() for request-response.

// app.module.ts
import { Module } from '@nestjs/common';
import { ClientsModule, Transport } from '@nestjs/microservices';
import { GatewayController } from './gateway.controller';

@Module({
  imports: [
    ClientsModule.register([
      {
        name: 'ORDERS_CLIENT',
        transport: Transport.KAFKA,
        options: {
          client: { clientId: 'gateway', brokers: ['kafka-1:9092'] },
          consumer: { groupId: 'gateway-consumer' },
        },
      },
    ]),
  ],
  controllers: [GatewayController],
})
export class AppModule {}

// gateway.controller.ts
import { Controller, Get, Inject, OnModuleInit, Post } from '@nestjs/common';
import { ClientKafka } from '@nestjs/microservices';
import { Observable } from 'rxjs';

@Controller('orders')
export class GatewayController implements OnModuleInit {
  constructor(
    @Inject('ORDERS_CLIENT') private readonly client: ClientKafka,
  ) {}

  async onModuleInit() {
    // Required for request-response: subscribe to reply topics before connecting.
    this.client.subscribeToResponseOf('order.created');
    await this.client.connect();
  }

  @Post()
  createOrder(): Observable<{ id: string; status: string }> {
    return this.client.send('order.created', { id: 'a1', total: 4200 });
  }

  @Get('ship')
  shipOrder() {
    this.client.emit('order.shipped', { id: 'a1' });
    return { accepted: true };
  }
}

The reply-topic pattern

Kafka has no built-in correlation for request-response, so NestJS layers it on. When you call send('order.created', ...), the framework publishes to order.created, attaches a generated correlation id, and waits for a record on order.created.reply. The consuming service’s @MessagePattern return value is published back to that reply topic. This is why you must call subscribeToResponseOf() for every request pattern in onModuleInit() before connect() — otherwise the client never joins the reply topic and the send() Observable hangs.

Prefer emit() over send() whenever you can. Request-response on Kafka adds a reply topic, extra partitions, and latency; events play to Kafka’s strengths.

Partitions and consumer groups

A topic is split into partitions, and ordering is only guaranteed within a single partition. Kafka routes a record to a partition by hashing its key, so messages sharing a key always land in the same partition and stay ordered relative to each other. Set the key through the message envelope when emitting.

this.client.emit('order.created', {
  key: order.customerId, // all events for one customer share a partition
  value: order,
});

The relationship between partitions, consumers, and groups drives scaling:

Concept	Role	Scaling rule
Partition	Unit of ordering and parallelism	More partitions = more parallel consumers
Consumer group (`groupId`)	Set of instances sharing the work	Each partition assigned to one member
Idle consumer	Member with no partition	Occurs when members > partitions

If a group has more instances than the topic has partitions, the extra instances sit idle. Provision enough partitions up front to match your target consumer count, because increasing partitions later changes key-to-partition mapping and breaks per-key ordering for in-flight data.

Best Practices

Always set a stable groupId; instances sharing it scale out automatically and avoid double-processing.
Make @EventPattern handlers idempotent — Kafka redelivers on handler failure and at-least-once is the default.
Call subscribeToResponseOf() for every send() pattern in onModuleInit() before connect().
Use a message key to pin related events to one partition and preserve ordering.
Size partitions to your peak consumer count from the start to avoid idle members and rebalancing churn.
Favor emit() (events) over send() (request-response) to stay aligned with Kafka’s streaming model.
Tune consumer options (sessionTimeout, heartbeatInterval, maxBytesPerPartition) for your throughput rather than relying on KafkaJS defaults.