How to Run Background Jobs in Node.js Without Setting Up Infrastructure

By Muhammad Zain8 min read
backendnodejsapiarchitecture

Your API endpoint needs to do three things when a user signs up: create the account, send a welcome email, and resize their profile photo. You could do all three synchronously, but now your /api/register endpoint takes 8 seconds to respond because the email service is slow and image processing blocks the event loop.

So you think: "I need a background job queue." Then you look at the options: RabbitMQ needs a server. AWS SQS needs IAM configuration. Kafka is enterprise-level overkill. Suddenly a simple "send email in background" task feels like a DevOps project.

It doesn't have to be. Let me walk you through three approaches — from zero-dependency to production-grade — and when to use each.

Why You Can't Just Use setTimeout

// ❌ This is not a background job
app.post('/api/register', async (req, res) => {
  const user = await createUser(req.body);
  res.json({ success: true }); // Respond immediately
 
  // "Background" work
  setTimeout(async () => {
    await sendWelcomeEmail(user.email);
    await resizeProfilePhoto(user.id);
  }, 0);
});

This looks like it works, but:

  1. No retry on failure. If sendWelcomeEmail throws, the email is permanently lost.
  2. No persistence. If the server crashes or restarts, pending jobs vanish.
  3. No concurrency control. 1,000 signups = 1,000 simultaneous image resize operations hammering your CPU.
  4. No observability. You can't see what's pending, what failed, or what completed.

setTimeout is fire-and-forget. Real background jobs need retry logic, persistence, and monitoring.

Approach 1: In-Process Queue (Zero Dependencies)

For simple use cases — sending emails, logging analytics events, or fire-and-forget tasks — a lightweight in-process queue with retry logic is all you need.

// lib/simple-queue.ts
 
type Job<T> = {
  id: string;
  data: T;
  attempts: number;
  maxAttempts: number;
  createdAt: Date;
};
 
type JobHandler<T> = (data: T) => Promise<void>;
 
export class SimpleQueue<T> {
  private queue: Job<T>[] = [];
  private processing = false;
  private handler: JobHandler<T>;
  private concurrency: number;
  private activeCount = 0;
 
  constructor(handler: JobHandler<T>, concurrency = 3) {
    this.handler = handler;
    this.concurrency = concurrency;
  }
 
  add(data: T, maxAttempts = 3): string {
    const id = crypto.randomUUID();
    this.queue.push({
      id,
      data,
      attempts: 0,
      maxAttempts,
      createdAt: new Date(),
    });
    this.process();
    return id;
  }
 
  private async process(): Promise<void> {
    if (this.processing) return;
    this.processing = true;
 
    while (this.queue.length > 0 && this.activeCount < this.concurrency) {
      const job = this.queue.shift();
      if (!job) break;
 
      this.activeCount++;
      this.executeJob(job).finally(() => {
        this.activeCount--;
        if (this.queue.length > 0) this.process();
      });
    }
 
    this.processing = false;
  }
 
  private async executeJob(job: Job<T>): Promise<void> {
    job.attempts++;
    try {
      await this.handler(job.data);
      console.log(`[Queue] Job ${job.id} completed`);
    } catch (error) {
      console.error(`[Queue] Job ${job.id} failed (attempt ${job.attempts}/${job.maxAttempts})`);
      if (job.attempts < job.maxAttempts) {
        // Exponential backoff: 1s, 2s, 4s
        const delay = Math.pow(2, job.attempts - 1) * 1000;
        setTimeout(() => {
          this.queue.push(job);
          this.process();
        }, delay);
      } else {
        console.error(`[Queue] Job ${job.id} permanently failed`, error);
        // Optionally: save to a dead letter store
      }
    }
  }
 
  get pendingCount(): number {
    return this.queue.length;
  }
 
  get activeJobs(): number {
    return this.activeCount;
  }
}

Using It

// queues/email-queue.ts
import { SimpleQueue } from '@/lib/simple-queue';
import { sendEmail } from '@/lib/email';
 
type EmailJob = {
  to: string;
  subject: string;
  html: string;
};
 
export const emailQueue = new SimpleQueue<EmailJob>(async (job) => {
  await sendEmail(job.to, job.subject, job.html);
}, 5); // Process 5 emails at a time
 
// In your API route:
app.post('/api/register', async (req, res) => {
  const user = await createUser(req.body);
 
  // Queue email — returns immediately
  emailQueue.add({
    to: user.email,
    subject: 'Welcome!',
    html: renderWelcomeEmail(user.name),
  });
 
  res.json({ success: true });
});

When this is enough:

  • Tasks that can be lost on server restart (emails can be re-triggered)
  • Low-volume processing (< 100 jobs/minute)
  • Single server deployment

When you need more: Multiple servers, tasks that must survive restarts, or heavy processing.

Approach 2: BullMQ + Redis (Production-Grade)

BullMQ is the go-to solution for Node.js background jobs. It uses Redis for persistence, supports retries, priorities, scheduling, and rate limiting — and it's surprisingly easy to set up.

npm install bullmq

Define the Queue and Worker

// queues/image-queue.ts
import { Queue, Worker } from 'bullmq';
 
const connection = {
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
};
 
// Queue: where you ADD jobs
export const imageQueue = new Queue('image-processing', { connection });
 
// Worker: where jobs are PROCESSED
const worker = new Worker('image-processing', async (job) => {
  const { userId, imagePath } = job.data;
 
  console.log(`Processing image for user ${userId}`);
 
  // Step 1: Download original
  const original = await downloadImage(imagePath);
 
  // Step 2: Generate sizes
  await Promise.all([
    generateThumbnail(original, userId, 150),   // 150px thumb
    generateMedium(original, userId, 600),      // 600px medium
    generateLarge(original, userId, 1200),       // 1200px large
  ]);
 
  // Step 3: Update user record
  await updateUserProfileImage(userId, {
    thumbnail: `/images/${userId}/thumb.webp`,
    medium: `/images/${userId}/medium.webp`,
    large: `/images/${userId}/large.webp`,
  });
 
  // Progress tracking
  await job.updateProgress(100);
}, {
  connection,
  concurrency: 3,          // Process 3 images simultaneously
  limiter: {
    max: 10,               // Max 10 jobs
    duration: 60_000,      // Per minute (rate limiting)
  },
});
 
worker.on('completed', (job) => {
  console.log(`Image job ${job.id} completed`);
});
 
worker.on('failed', (job, err) => {
  console.error(`Image job ${job?.id} failed:`, err.message);
});

Adding Jobs

// In your API route
app.post('/api/profile/photo', async (req, res) => {
  const { userId, imagePath } = req.body;
 
  await imageQueue.add('resize', {
    userId,
    imagePath,
  }, {
    attempts: 3,                              // Retry up to 3 times
    backoff: { type: 'exponential', delay: 2000 }, // 2s, 4s, 8s
    removeOnComplete: { count: 100 },         // Keep last 100 completed
    removeOnFail: { count: 500 },             // Keep last 500 failed
    priority: 1,                              // Higher priority
  });
 
  res.json({ message: 'Photo processing started' });
});

Scheduled / Delayed Jobs

// Send follow-up email 24 hours after signup
await emailQueue.add('follow-up', {
  userId: user.id,
  template: 'day-one-tips',
}, {
  delay: 24 * 60 * 60 * 1000, // 24 hours
});
 
// Run a cleanup job every hour
await maintenanceQueue.add('cleanup', {}, {
  repeat: {
    every: 60 * 60 * 1000, // Every hour
  },
});

Monitoring with Bull Board

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';
 
const serverAdapter = new ExpressAdapter();
createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(imageQueue),
  ],
  serverAdapter,
});
 
app.use('/admin/queues', serverAdapter.getRouter());

Now you have a dashboard at /admin/queues showing all pending, active, completed, and failed jobs.

Approach 3: Serverless-Friendly (Vercel, Cloudflare, AWS Lambda)

On serverless platforms, you can't run persistent workers. The function executes and dies. You have two options:

Option A: waitUntil (Vercel / Cloudflare)

// app/api/register/route.ts (Next.js)
import { after } from 'next/server';
 
export async function POST(request: Request) {
  const user = await createUser(await request.json());
 
  // This runs AFTER the response is sent
  after(async () => {
    await sendWelcomeEmail(user.email);
    await trackAnalyticsEvent('user_signup', user.id);
  });
 
  return Response.json({ success: true });
}

after() (Next.js 15+) lets you run code after the response is sent. The function invocation stays alive until the work completes. It's not a queue — there's no retry — but it's perfect for non-critical tasks.

Option B: External Queue Service

For reliable serverless background jobs, use a hosted queue:

  • Upstash QStash — HTTP-based message queue built for serverless
  • Inngest — Event-driven background functions
  • Trigger.dev — Background jobs with built-in retry and scheduling
// Using QStash
import { Client } from '@upstash/qstash';
 
const qstash = new Client({ token: process.env.QSTASH_TOKEN! });
 
// In your API route — queue the job
await qstash.publishJSON({
  url: 'https://yourapp.com/api/jobs/send-email',
  body: { userId: user.id, template: 'welcome' },
  retries: 3,
  delay: 0,
});
 
// The job endpoint processes it
// app/api/jobs/send-email/route.ts
export async function POST(request: Request) {
  const { userId, template } = await request.json();
  await sendTemplatedEmail(userId, template);
  return Response.json({ success: true });
}

QStash calls your endpoint via HTTP, with built-in retries and dead letter queues. Your serverless function doesn't stay alive — QStash triggers a new invocation for each job.

Which Approach to Choose

| Scenario | Approach | Why | |----------|----------|-----| | Single server, low volume | In-process queue | Zero dependencies, simple | | Multiple servers, high volume | BullMQ + Redis | Persistent, scalable, monitored | | Serverless (Vercel/Cloudflare) | after() or QStash | No persistent process needed | | Critical jobs (payments) | BullMQ or external service | Must survive crashes | | Quick fire-and-forget (analytics) | after() or in-process | Doesn't need persistence |

Key Takeaways

  • setTimeout is not a background job. No retry, no persistence, no monitoring.
  • In-process queues work for single-server, non-critical tasks.
  • BullMQ is the standard for Node.js apps that need reliability and observability.
  • Serverless needs external triggers. Use after(), QStash, or Inngest.
  • Always add retry logic. External services fail. Your jobs should retry automatically.
  • Monitor your queues. Failed jobs that nobody notices are worse than no jobs at all.

Background jobs are one of those things every backend developer needs but nobody teaches in tutorials. Start simple, add complexity as your scale demands it, and always make sure you can see what's happening in your queues.

Find me at: itszain.dev

Related Articles

How to Build a REST API in Node.js That Won't Embarrass You in a Code Review

Most Node.js API tutorials teach bad patterns — no validation, no error handling, hardcoded status codes, and business logic in route handlers. Here's how to structure a production-grade REST API with Express that senior engineers will respect.

11 min read

Your Next.js API Routes Are Wide Open — How to Add Rate Limiting Before You Get Abused

Most Next.js apps deploy API routes with zero protection against abuse. Here's how to add production-grade rate limiting using in-memory stores, Redis, and middleware — before someone hammers your endpoints.

8 min read

How to Show Real File Upload Progress in Flutter (Not Fake Progress Bars)

Most Flutter upload tutorials fake the progress bar. Here's how to track actual byte-level upload progress using Dio's multipart requests and stream transformers — with a production-ready implementation.

5 min read