Message Brokers & Async Messaging

Synchronous HTTP couples services in time: the caller blocks until the callee answers, and if the callee is down the call fails. Asynchronous messaging breaks that coupling. A producer publishes an event to a message broker and moves on; one or more consumers process it later, at their own pace. This is the backbone of event-driven architecture — it absorbs traffic spikes, lets services fail and recover independently, and turns a fragile chain of HTTP calls into a resilient pipeline. This page covers the two dominant brokers, RabbitMQ and Apache Kafka, from an Express producer and consumer.

When to reach for messaging

Use a broker when work can happen after the HTTP response, when several services care about the same event, or when you need to smooth bursty load. A classic example: an orders service confirms a checkout instantly, then emits an order.placed event. Email, inventory, and analytics services each consume it without the orders service knowing they exist.

Pattern	Synchronous HTTP	Async messaging
Coupling	Caller waits for callee	Fire-and-forget
Failure	Callee down → call fails	Broker buffers until consumer recovers
Fan-out	Caller loops over services	One publish, many subscribers
Load spikes	Backpressure hits the caller	Broker queue absorbs the burst

Tip: Messaging is not a drop-in replacement for request/response. If the client genuinely needs the answer now (e.g. “is this username taken?”), keep it synchronous. Use events for work that can complete out of band.

Queues vs topics

The two brokers model delivery differently, and the distinction drives your design.

RabbitMQ (queues + exchanges) — a producer publishes to an exchange, which routes the message to one or more queues based on a routing key. Each message in a queue is delivered to exactly one of the competing consumers, so adding consumers spreads the load (a work queue). A fanout exchange copies a message to every bound queue for pub/sub.
Kafka (topics + partitions) — producers append events to a topic, which is split into ordered partitions. Consumers join a consumer group; each partition is read by one consumer in the group. Events are retained on disk for a configured period, so new or replaying consumers can re-read history. This makes Kafka a durable log, not just a transient queue.

	RabbitMQ	Kafka
Model	Smart broker, dumb consumer	Dumb broker, smart consumer
Retention	Removed once acked	Retained by time/size, replayable
Ordering	Per queue	Per partition
Scaling unit	Competing consumers	Partitions per group
Best for	Task queues, RPC, routing	Event streaming, high throughput, replay

Producing and consuming with RabbitMQ

Install the official client and connect once at startup. The producer asserts a durable queue and publishes persistent messages so they survive a broker restart.

npm install amqplib express

const amqp = require('amqplib');

let channel;
async function connect() {
  const conn = await amqp.connect(process.env.AMQP_URL); // amqp://localhost
  channel = await conn.createChannel();
  await channel.assertQueue('order.placed', { durable: true });
}

// Producer: publish from an Express route
app.post('/orders', express.json(), async (req, res) => {
  const order = { id: crypto.randomUUID(), ...req.body };
  channel.sendToQueue(
    'order.placed',
    Buffer.from(JSON.stringify(order)),
    { persistent: true }, // write to disk, survive restart
  );
  res.status(202).json({ accepted: true, orderId: order.id });
});

The consumer runs as a separate worker process. prefetch(1) limits it to one un-acked message at a time so work is distributed fairly, and it only acks after the job succeeds.

async function startWorker() {
  const conn = await amqp.connect(process.env.AMQP_URL);
  const channel = await conn.createChannel();
  await channel.assertQueue('order.placed', { durable: true });
  await channel.prefetch(1);

  channel.consume('order.placed', async (msg) => {
    if (!msg) return;
    const order = JSON.parse(msg.content.toString());
    try {
      await sendConfirmationEmail(order);
      channel.ack(msg);                 // success: remove from queue
    } catch (err) {
      channel.nack(msg, false, true);   // failure: requeue for retry
    }
  });
}

Output:

POST /orders → 202 Accepted
[worker] consumed order.placed { id: "3f2a…", total: 4200 }
[worker] confirmation email sent, message acked

Producing and consuming with Kafka

For Kafka, kafkajs is the standard Node client. A producer sends to a topic; using the order ID as the message key guarantees all events for one order land on the same partition and stay ordered.

npm install kafkajs express

const { Kafka } = require('kafkajs');
const kafka = new Kafka({ clientId: 'orders', brokers: ['localhost:9092'] });

const producer = kafka.producer();
await producer.connect();

app.post('/orders', express.json(), async (req, res) => {
  const order = { id: crypto.randomUUID(), ...req.body };
  await producer.send({
    topic: 'order.placed',
    messages: [{ key: order.id, value: JSON.stringify(order) }],
  });
  res.status(202).json({ accepted: true, orderId: order.id });
});

A consumer joins a group. Kafka commits the offset after eachMessage resolves, so throwing keeps the offset un-advanced and the message is redelivered — the foundation of at-least-once delivery.

const consumer = kafka.consumer({ groupId: 'email-service' });
await consumer.connect();
await consumer.subscribe({ topic: 'order.placed', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ message }) => {
    const order = JSON.parse(message.value.toString());
    await sendConfirmationEmail(order); // throw → offset not committed → redelivered
  },
});

At-least-once delivery and idempotency

Both brokers default to at-least-once delivery: if a consumer crashes after doing work but before acking (RabbitMQ) or committing the offset (Kafka), the message is redelivered. That means consumers will occasionally see duplicates. The fix is to make handlers idempotent — processing the same event twice has the same effect as once.

async function handleOrderPlaced(order) {
  // dedupe on a unique key; skip if already processed
  const inserted = await db.query(
    `INSERT INTO processed_events (event_id) VALUES ($1)
     ON CONFLICT (event_id) DO NOTHING RETURNING event_id`,
    [order.id],
  );
  if (inserted.rowCount === 0) return; // duplicate, ignore
  await sendConfirmationEmail(order);
}

Warning: Never assume exactly-once delivery from the broker alone. Even Kafka’s transactional producer only guarantees exactly-once within Kafka; the moment you call an external API or database, you own deduplication. Design every consumer to tolerate replays.

Best Practices

Treat the broker as the source of truth for in-flight work: publish persistent/durable messages and only ack after the job actually succeeds.
Make every consumer idempotent — dedupe on an event ID so redeliveries are harmless.
Run consumers as separate worker processes, not inside the HTTP server, so request latency and background work scale independently.
Set a prefetch/concurrency limit (RabbitMQ prefetch, Kafka partition count) to control how much each consumer pulls at once.
Route poison messages that repeatedly fail to a dead-letter queue/topic instead of requeuing them forever.
Use a Kafka message key (or RabbitMQ routing key) deliberately when ordering matters; without it, events fan out across partitions and lose order.
Version your event payloads and keep them backward-compatible so producers and consumers can deploy independently.