Project: Background Job System

Some work has no business blocking an HTTP request: sending email, generating PDFs, transcoding video, or calling slow third-party APIs. The fix is to push that work onto a queue, return immediately, and let a separate pool of workers grind through it with retries, backoff, and scheduling. In this project you’ll build a background job system in NestJS with BullMQ — a Redis-backed queue that gives you durable jobs, concurrency control, recurring schedules, and a dashboard to watch it all run.

How BullMQ fits together

BullMQ has three moving parts. A Queue is the producer-side handle you use to add jobs. A Worker is a long-running process that pulls jobs off the queue and runs your handler. Redis sits in the middle as durable storage, so jobs survive a crash. NestJS wraps all of this with @nestjs/bullmq, which registers queues as injectable providers and turns workers into decorated classes managed by the DI container.

npm install @nestjs/bullmq bullmq ioredis
# Redis must be running — locally:
docker run -d -p 6379:6379 redis:7

Registering the queue module

Configure the Redis connection once with BullModule.forRoot, then register each named queue with BullModule.registerQueue. A named queue can be both produced into (in a controller) and consumed from (in a worker processor).

// app.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';
import { EmailModule } from './email/email.module';

@Module({
  imports: [
    BullModule.forRoot({
      connection: { host: 'localhost', port: 6379 },
    }),
    EmailModule,
  ],
})
export class AppModule {}

// email/email.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';
import { EmailController } from './email.controller';
import { EmailProcessor } from './email.processor';

@Module({
  imports: [BullModule.registerQueue({ name: 'email' })],
  controllers: [EmailController],
  providers: [EmailProcessor],
})
export class EmailModule {}

Enqueuing jobs from an API

Inject the queue with @InjectQueue('email') and call add(). The first argument is the job name (workers can branch on it), the second is the payload, and the third configures retries and backoff. Returning the job id lets clients poll status later.

// email/email.controller.ts
import { Body, Controller, Post } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';

interface SendDto {
  to: string;
  subject: string;
}

@Controller('email')
export class EmailController {
  constructor(@InjectQueue('email') private readonly queue: Queue) {}

  @Post('send')
  async send(@Body() dto: SendDto) {
    const job = await this.queue.add('welcome', dto, {
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: 1000,
      removeOnFail: 5000,
    });
    return { jobId: job.id, status: 'queued' };
  }
}

Set removeOnComplete/removeOnFail to a number or true in production. Without them, finished jobs accumulate in Redis forever and slowly exhaust memory.

Processing jobs in a worker

Extend WorkerHost and decorate the class with @Processor. The process method runs for every job; switch on job.name to handle multiple job types in one queue. Set concurrency to control how many jobs run in parallel per worker process. Throwing an error marks the job failed and triggers the backoff/retry policy you configured at enqueue time.

// email/email.processor.ts
import { Processor, WorkerHost, OnWorkerEvent } from '@nestjs/bullmq';
import { Logger } from '@nestjs/common';
import { Job } from 'bullmq';

@Processor('email', { concurrency: 5 })
export class EmailProcessor extends WorkerHost {
  private readonly logger = new Logger(EmailProcessor.name);

  async process(job: Job): Promise<{ delivered: boolean }> {
    this.logger.log(`Processing ${job.name} #${job.id} (attempt ${job.attemptsMade + 1})`);

    if (job.name === 'welcome') {
      await this.deliver(job.data.to, job.data.subject);
      return { delivered: true };
    }
    throw new Error(`Unknown job: ${job.name}`);
  }

  private async deliver(to: string, subject: string): Promise<void> {
    // Real transport call (SMTP, SES, etc.)
    await new Promise((r) => setTimeout(r, 200));
    this.logger.log(`Sent "${subject}" to ${to}`);
  }

  @OnWorkerEvent('failed')
  onFailed(job: Job, err: Error) {
    this.logger.error(`Job ${job.id} failed: ${err.message}`);
  }
}

Output:

[Nest] LOG [EmailProcessor] Processing welcome #1 (attempt 1)
[Nest] LOG [EmailProcessor] Sent "Welcome!" to [email protected]

Scheduling recurring jobs

BullMQ supports repeatable jobs driven by a cron expression or a fixed interval. Add them once at startup (BullMQ deduplicates by repeat key, so re-adding on every boot is safe). This is ideal for nightly cleanups, report generation, or polling an upstream system.

// reports/reports.scheduler.ts
import { Injectable, OnModuleInit } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';

@Injectable()
export class ReportsScheduler implements OnModuleInit {
  constructor(@InjectQueue('email') private readonly queue: Queue) {}

  async onModuleInit() {
    await this.queue.add(
      'daily-digest',
      { type: 'digest' },
      { repeat: { pattern: '0 8 * * *' }, jobId: 'daily-digest' },
    );
  }
}

Retries, backoff, and delays

The add() options control resilience. A few you’ll reach for often:

Option	Purpose
`attempts`	Total tries before the job is marked failed
`backoff`	`fixed` or `exponential` delay between retries
`delay`	Milliseconds to wait before the job becomes runnable
`priority`	Lower number = processed sooner
`jobId`	Custom id; deduplicates repeated enqueues

A failing job that exhausts its attempts lands in the failed set. Build a small endpoint that calls queue.getFailed() and job.retry() so operators can replay them after a fix.

Monitoring the queue

For visibility, mount the Bull Board dashboard. It shows waiting, active, completed, and failed jobs with payloads and stack traces.

npm install @bull-board/api @bull-board/express

// main.ts (excerpt)
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/admin/queues');
createBullBoard({
  queues: [new BullMQAdapter(emailQueue)],
  serverAdapter,
});
app.use('/admin/queues', serverAdapter.getRouter());

Best practices

Keep job payloads small — store an id and re-fetch the record in the worker rather than serializing large objects into Redis.
Make handlers idempotent; a job can run more than once after a crash or retry, so guard against duplicate side effects.
Set explicit attempts and backoff on every job instead of relying on defaults, and use exponential backoff for anything that hits a network.
Always set removeOnComplete/removeOnFail to cap Redis memory growth.
Run workers as separate processes from your API so a flood of jobs never starves request handling.
Tune concurrency per workload — CPU-bound jobs want low concurrency, I/O-bound jobs can run many in parallel.
Protect the Bull Board dashboard behind authentication; it exposes raw job payloads.