A simple way to create a Node.JS Transform stream which processes in parallel. You can limit the concurrency (default is 16) and order is not preserved (so chunks/objects can end up in a different order to the order they started in if the transform functions take different amounts of time).

Built using through2 and has the same API with the addition of a maxConcurrency option.

Non-objectMode streams are supported for completeness but I'm not sure they'd be useful for anything.

Written by Thomas Parslow (almostobsolete.net and tomparslow.co.uk) as part of Active Inbox (activeinboxhq.com).

npm install --save through2-concurrent

Process lines from a CSV in parallel. The order the results end up in the all variable is not deterministic.

var through2Concurrent = require('through2-concurrent');

var all = [];

fs.createReadStream('data.csv')
  .pipe(csv2())
  .pipe(through2Concurrent.obj(
    {maxConcurrency: 10},
    function (chunk, enc, callback) {
      var self = this;
      someThingAsync(chunk, function (newChunk) {
        self.push(newChunk);
        callback();
      });
  }))
  .on('data', function (data) {
    all.push(data)
  })
  .on('end', function () {
    doSomethingSpecial(all)
  })

Fixed or improved stuff? Great! Send me a pull request through GitHub or get in touch on Twitter @almostobsolete or email at [email protected]