There are two apps in this project. A producer and a consumer/worker app. This project shows you how RabbitMQ smartly schedules tasks sent by a producer to various (Node.js) workers. This project also shows 'ack' (acknowledgement) feature.
Specifically it shows:
- RabbitMQ schedules tasks in round-robin fashion among workers
- W/ ACK, If one of the worker takes long time (long job or is hanging), RabbitMQ spreads the work to other AVAILABLE workers.
- W/ ACK, If one of the worker dies while processing a job, RabbitMQ reschedules the job.
- Most importantly, how you can implement all these w/ just few lines of code!
- cd to RabbitWorkers/worker folder
- `npm install`
- Push the app to cloud foundry as `standalone` (background app) app w/ 4 instances & connect to RabbitMQ service w/ name `rabbit-service1`
vmc push rabbitworker
Instances> 4
1: node
2: other
Framework> 2
1: sinatra
2: rack
3: lift
4: grails
5: rails3
6: spring
7: java_web
8: standalone
9: node
10: play
Framework> 8
Startup command> node app.js
1: node
2: node06
3: node08
4: other
Runtime> 3
1: 64M
2: 128M
3: 256M
4: 512M
Memory Limit> 64M
Creating rabbitworker... OK
1: rabbitworker.cloudfoundry.com
2: none
URL> 2
Create services for application?> y
1: blob 0.51
2: mongodb 2.0
3: mysql 5.1
4: postgresql 9.0
5: rabbitmq 2.4
6: redis 2.6
7: redis 2.4
8: redis 2.2
What kind?> 5
Name?> rabbit-service1
Creating service rabbit-service1... OK
Binding rabbit-service1 to rabbitworker... OK
Create another service?> n
Bind other services to application?> n
Save configuration?> n
Uploading rabbitworker... OK
Starting rabbitworker... OK
Checking rabbitworker... OK
- cd to RabbitWorkers/producer folder
- `npm install`
- Push the app to cloud foundry as `standalone` (background app) app w/ RabbitMQ service `rabbit-service1`
> vmc push rabbitproducer
Instances> 1
1: node
2: other
Framework> 2
1: node
2: lift
3: rails3
4: sinatra
5: play
6: java_web
7: grails
8: standalone
9: rack
10: spring
Framework> 8
Startup command> node app.js
1: node
2: node06
3: node08
4: other
Runtime> 3
1: 64M
2: 128M
3: 256M
4: 512M
Memory Limit> 64M
Creating rabbitproducer... OK
1: rabbitproducer.cloudfoundry.com
2: none
URL> 2
Create services for application?> n
Bind other services to application?> y
1: rabbit-service1
2: mongodb-foobar
Which service?> 1
Binding rabbit-service1 to rabbitproducer... OK
Bind another service?> n
Save configuration?> n
Uploading rabbitproducer... OK
Starting rabbitproducer... OK
Checking rabbitproducer... OK
- Wait for 10 second for the job to be processed. Then check logs:
vmc logs rabbitworker --all Getting logs for rabbitworker #0... OK
Reading logs/stderr.log... OK
Reading logs/stdout.log... OK connected to RabbitMQ Got task: Task# 1
Now processing batch: impartial-lunchroom
Total tasks processed: 1 Got task: Task# 5 Total tasks processed: 2 Got task: Task# 10 Total tasks processed: 3 Got task: Task# 11 Total tasks processed: 4 Got task: Task# 13 Total tasks processed: 5 Got task: Task# 18 Total tasks processed: 6 Got task: Task# 20 Total tasks processed: 7
Getting logs for rabbitworker #1... OK
Reading logs/stderr.log... OK
Reading logs/stdout.log... OK connected to RabbitMQ Got task: Task# 2
Now processing batch: impartial-lunchroom
Total tasks processed: 1 Got task: Task# 7 Total tasks processed: 2 Got task: Task# 8 Total tasks processed: 3 Got task: Task# 12 Total tasks processed: 4 Got task: Task# 15 Total tasks processed: 5 Got task: Task# 16 Total tasks processed: 6 Got task: Task# 19 Total tasks processed: 7
Getting logs for rabbitworker #2... OK
Reading logs/stderr.log... OK
Reading logs/stdout.log... OK connected to RabbitMQ Got task: Task# 3
Now processing batch: impartial-lunchroom
Getting logs for rabbitworker #3... OK
Reading logs/stderr.log... OK
Reading logs/stdout.log... OK connected to RabbitMQ Got task: Task# 4
Now processing batch: impartial-lunchroom
Total tasks processed: 1 Got task: Task# 6 Total tasks processed: 2 Got task: Task# 9 Total tasks processed: 3 Got task: Task# 3 <--- (redelivered) Total tasks processed: 4 Got task: Task# 14 Total tasks processed: 5 Got task: Task# 17 Total tasks processed: 6
<h4> Play around w/ `doTask` function in `worker/app.js` to simulate long jobs and worker-crashes</h4>
```javascript
function doTask(q, task, redelivered) {
//default time for task 1 second
var taskTime = 1000;
//Comment the below section to make all jobs take 1 second.
//Task #4 takes 5 seconds (instead of typical 1 sec)
if (task.taskName === 'Task# 4') {
taskTime = 5000;
}
updateTaskCount(task, redelivered);
// Comment the below section to spread job evenly
//Crash the worker if it gets task #3
// and a not-redelivered(otherwise, we will crash all instance when are redelivered by RabbitMQ)
if (task.taskName === 'Task# 3' && !redelivered) {
q.close();
return;
}
//Each task takes 1 second to process
setTimeout(function () {
console.log('Total tasks processed: ' + batchNameAndCnt[task.batchName]);
q.shift(); // MAKE SURE TO DO SHIFT (to send ACK)
}, taskTime);
}
- Install RabbitMQ
- Simply open at least 2 terminal windows for workers and 1 more for producer.
- cd to rabbitworker/worker in worker terminals, do `npm install` and run `node app.js`. And keep them running.
- cd to rabbitworker/producer do `npm install` and run `node app.js`
- Check the console.log outputs in worker terminal windows.
- Play around w/ `doTask` function in `worker/app.js` to simulate long jobs and worker-crashes