nodejs-background-worker's Introduction

NodeJS Background Worker

Introduction

This repository is a demonstration of actioning background processing / jobs using Bull, a Redis-based queue for Node. In this example we create a job queue in Redis, write jobs into said queue, and attach a worker to process said jobs. There are three containers;

A redis container that persists its data to disk (in redis-data/) to ensure jobs live between container teardowns and restarts.
A service container using Express that hosts a dashboard and an endpoint to create and place jobs into a queue.
A worker container that attaches a processor function to the queue.

Instructions

These instructions assumes you already have docker and docker-compose installed. The steps are as follows:

Run the service which will also bring up the redis container and create the queue (should it not already exist yet).

docker-compose up service

Hit the jobs endpoint a multitude of times to queue up dummy jobs, ready for processing.

curl -X POST http://localhost:8080/job

Open bull-board dashboard and see the "waiting" tab is populated with the jobs you spun up. They will remain here until we start the worker and get processed.
Start the worker in another tab / terminal.

docker-compose up worker

Observe that jobs are picked up from "waiting", moved to "active", progressed over time then moved to "completed". You will also see appropriate logging in the workers output. Jobs have been given an artificial 50% chance to fail, so you can potentially see them be attempted multiple times, and moved to "delayed" as they are exponentially backed off and attempted up to a maximum number of times, before being moved to "failed".