Fairy is a lightweight queue engine for node.js based on Redis. Fairy offers ActiveMQ's message groups alike feature which can guarantee the sequential processing order of tasks belong to a same group.
But, unlike message groups, Fairy doesn't always route tasks of a group to a same worker, which will introduce unwanted waiting time when:
- Tasks of group
XandYare appointed to workerA. - Worker
Ais processing tasks of groupXsequentially. - Tasks of group
Yare pending, while: - Worker
Bis still idling because of 1.
Fairy will route the task of group Y to worker B in this scenario.
Fairy takes a different approach than Message Groups. Instead of making all tasks of a same group be routed to the same consumer, Fairy route a task to any worker when there's no processing tasks of the same group.
The design philosophy makes Fairy ideal for the following requirements:
- Tasks of a same groups need be processed in sequence.
- Each worker processes tasks in serial.
- Multiple workers need be instantiated to increase throughput.
Fairy takes a different approach than Message Groups. Instead of making all tasks of a same group be routed to the same consumer, Fairy route a task to any worker when there's no processing tasks of the same group.
When the number of workers is much smaller compared to the number of groups, Fairy's approach makes sense.
Resque cannot guarantee the processing order of the tasks although the task queue is FIFO. The more workers you have, the more possible you'll encountering concurrency which breaks the processing order of tasks in the same group.
npm install fairy
The minimium set of APIs you need to learn in order to implement a task queue system are:
enqueuetasks, andregista function for processing them.
Provide as many parameters as you want (and an optional callback function). The first argument will be used for message grouping.
queue = require('fairy').connect().queue('task_name')
queue.enqueue 'foo', 'bar', ->
console.log 'your order has been placed, sir.'
When registered a task handler, the Fairy queue becomes a worker automatically.
The registered handler function will be called when there're tasks to be processed, with the enqueued parameters of the task. The last argument will be a non-optional callback function.
Arguments of the callback function follow node.js error handling convention:
err and res.
Calling the callback function is your responsibility (or Fairy will not dispatch tasks to the worker and block tasks of the same group forever!)
queue = require('fairy').connect().queue('task_name')
queue.regist (param1, param2, callback) ->
# Do your work here, be it synchronous or asynchronous.
callback err, res
Fairy comes with a ready-to-use web front-end. Simply insert the middleware into the pipeline:
app = require('express').createServer()
fairy_web = require 'fairy/web'
app.use fairy_web.connect().middleware
app.listen 3000
More APIs include:
- Objects of Class
Queue:
- Placing tasks --
enqueue - Regist handlers --
regist - Reschedule tasks --
reschedule - Query status --
recently_finished_tasksfailed_tasksblocked_groupsslowest_tasksprocessing_tasksworkersstatistics, etc.
- Objects of Class
Fairy:
queues, return all queues.statistics, return statistics of all queues.
See Example Folder for demos. Or review the Annotated Source Code for complete API explanations.
Copyright (c) 2012 Baoshan Sheng
Released under the MIT license.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent