creationix / step Goto Github PK

View Code? Open in Web Editor NEW

2.2K 84.0 188.0 29 KB

An async control-flow library that makes stepping through logic easy.

License: MIT License

JavaScript 100.00%

step's Introduction

Step

A simple control-flow library for node.JS that makes parallel execution, serial execution, and error handling painless.

How to install

Simply copy or link the lib/step.js file into your $HOME/.node_libraries folder.

How to use

The step library exports a single function I call Step. It accepts any number of functions as arguments and runs them in serial order using the passed in this context as the callback to the next step.

Step(
  function readSelf() {
    fs.readFile(__filename, this);
  },
  function capitalize(err, text) {
    if (err) throw err;
    return text.toUpperCase();
  },
  function showIt(err, newText) {
    if (err) throw err;
    console.log(newText);
  }
);

Notice that we pass in this as the callback to fs.readFile. When the file read completes, step will send the result as the arguments to the next function in the chain. Then in the capitalize function we're doing synchronous work so we can simple return the new value and Step will route it as if we called the callback.

The first parameter is reserved for errors since this is the node standard. Also any exceptions thrown are caught and passed as the first argument to the next function. As long as you don't nest callback functions inline your main functions this prevents there from ever being any uncaught exceptions. This is very important for long running node.JS servers since a single uncaught exception can bring the whole server down.

Also there is support for parallel actions:

Step(
  // Loads two files in parallel
  function loadStuff() {
    fs.readFile(__filename, this.parallel());
    fs.readFile("/etc/passwd", this.parallel());
  },
  // Show the result when done
  function showStuff(err, code, users) {
    if (err) throw err;
    console.log(code);
    console.log(users);
  }
)

Here we pass this.parallel() instead of this as the callback. It internally keeps track of the number of callbacks issued and preserves their order then giving the result to the next step after all have finished. If there is an error in any of the parallel actions, it will be passed as the first argument to the next step.

Also you can use group with a dynamic number of common tasks.

Step(
  function readDir() {
    fs.readdir(__dirname, this);
  },
  function readFiles(err, results) {
    if (err) throw err;
    // Create a new group
    var group = this.group();
    results.forEach(function (filename) {
      if (/\.js$/.test(filename)) {
        fs.readFile(__dirname + "/" + filename, 'utf8', group());
      }
    });
  },
  function showAll(err , files) {
    if (err) throw err;
    console.dir(files);
  }
);

Note that we both call this.group() and group(). The first reserves a slot in the parameters of the next step, then calling group() generates the individual callbacks and increments the internal counter.

step's People

Contributors

Stargazers

Watchers

Forkers

bryanwb nex3 simplegeo guybrush kristsk crabdude lordnox astro akidee xavi- phegaro stuarts codeinvain am9u coreyjewett fbrandel benjamine andykram ztrix sethml xuyuanok tinnt sequoiar josephconnolly friedmag scottgonzalez thdtjsdn youngrok xulrunner42 thedeveloper jvanalst starsun pkmishra crrlab marvinjeng strk dineshkummarc bf thanigaiv jjmiranda manirana007 livedata lovemfq ygm125 seanjsong diogogmt raffecat whitenode kaven276 gotomypc ricocai tinganho jkeylu codekick xuzhang aoj xrymz ejmvar prabirshrestha eugenionr shakakira hfeeki ryuever haowu eitanpo hguillermo jpillora sabarimanoj thisme pkarc alextong shichao nvdnkpr luciany riplexus mrhoda lanwin denofiend joshbedo namhyunkim java4ever lin-tun zyoo jahon web5design imclab prodigeni kingwjb jpjp71 youaani 1o-1o troland forestkatsch-archive joaoafrmartins deveshsinghal22 zcfrank1st newguyusegit mauricionobrega farmanyu suhail-ansari

step's Issues

step is just no value, it's just a trick

the code example in step's readme

Step(
  function readDir() {
    fs.readdir(__dirname, *this*);   <--------
  },
  function readFiles(err, results) {
    if (err) throw err;
    // Create a new group
    * var group = this.group() *;     <--------
    results.forEach(function (filename) {
      if (/\.js$/.test(filename)) {
        fs.readFile(__dirname + "/" + filename, 'utf8', *group()*);   <--------
      }
    });
  },
  function showAll(err , files) {
    if (err) throw err;
    console.dir(files);
  }
);

can just use this more clear code instead, replace all (this, this.group, this.async) away, step jsut play a trick, it sell you something you just have own already.

var fs = require('fs');
fs.readdir(__dirname, readFiles);

function readFiles(err, results){
  if (err) throw err;
  // Create a new group
  async.map(results, function(filename, cb){   <-------- use async.map to finally get all async result
    if (/\.js$/.test(filename)) {
      fs.readFile(__dirname + "/" + filename, 'utf8', cb);
    }
  }, showAll);
}

function showAll(err, files){
  if (err) throw err;
  console.dir(files);
}

Returning multiple parameters in a callback that has been grouped

I have this code on node v6.3.1 with step 1.0.0

var Step = require('step');

Step(
	function one() {
		doThis(0, "0", this);
	},
	function two(err, number, string) {
		console.log(err, number, string);

		var group = this.group();
		for (var i=0; i<2; i++) {
			doThis(i, String(i), group());
		}
	},
	function three(err, numbers, strings) {
		console.log(err, numbers, strings);
		if (err) process.exit(1);
		process.exit(0);
	}
);

function doThis(number, string, callback) {
	callback(null, number, string);
}

I get the following output

null 0 '0'
undefined [ 0, 1 ] undefined

So one calls do this which successfully callbacks with 0 and '0' but then I put it into a loop and use this.group() and it can no longer return the second parameter.

the expected output would be

null 0 '0'
null [ 0, 1 ] [ '0', '1' ]

More sophisticated options (could be v0.0.4)

Provide an option where step ignores normal return values. This is necessary when you use step in CoffeeScript where the last expression inside a function is always returned. Usually you do not need to use the sync return value, so you can switch it off by:

step.async()

Many CoffeeScript users will appreciate this change.

Be able to pass the functions' context:

step(
    contextObject,
    function (e, next) {

        // ...
        next(e, 4)
    },
    function (e, result, next) {

        doAsync(next.parallel())
        doAsync(next.parallel())
    },
    function (e, r1, r2) {

        // ...
    }
)

If you omit the contextObject, "next" will still be the context of the calls but is passed as the last argument as well. Convention: The first argument of every step is the error, the number of further arguments is dynamic, the last argument is "next"

Result: There are no significant API changes, so nobody needs to change any code, but you have more sophisticated options to make step more useful.

I could send a patch to integrate this functionality.

Andi

Serial in a step function ?

I have been looking for a simple way to chain actions when looping over an array but did not found any. It could be great if we can simply do :

Step(
  function() {
    var serial = this.serial;
    arr.forEach(function(el) {
      asyncFunction(el, serial);
    });
  },
  function end() {
    // everything's done
  }
);

but I did not succeeded in hacking the lib so I just wrote this function for the moment:

  function arrayChain(arr, func, callback) {
    var actions = arr.map(function(el) { return function() { func(el) }; });
    actions.push(callback);
    Step.apply(this, actions);
  };

Do you think this feature is useful/possible ?

Broken code in README (node v0.6.12)

Hi!

Running the first example yields:

has no method 'toUpperCase'

Simple fix:

return text.toString('utf8').toUpperCase();

Regards,

Włodek Bzyl

Please throw actual Errors, not strings

When step throws an exception, it just throws the string, rather than an actual Error object. This is bad because when an error is thrown, there is no stack trace to work with, so it's difficult to figure out the source of the error.

throw new Error('whatever');

problem in 'throw arguments[0]'

I have to replace the line at number 38 with :

  if (arguments[0]) {
    // throw arguments[0];
    var sys = require("sys");
    console.log("Uncaught error: " + sys.inspect(arguments[0],true,10,true));
  }

otherwise, when I run all my nodeunit tests, all testcases passed ,but this statement still thrown exception.

Please release 1.0.0 to npm

The semantic versioning specification states:

Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable.

However, in step's case, this isn't really accurate. At a certain point a project has been around long enough that it should be considered stable. Any future changes can be made in semver-major releases. (Also consider - especially for a smaller library like step, semi-frequent semver-major changes aren't the end of the world. They just aren't.)

Please publish 1.0.0 to npm to indicate that step is at least a somewhat mature project.

Consider API for next major version of Step

https://gist.github.com/1524578#comments

skipping steps

Is it possible to implement conditional control flow? I would like to skip over a few steps.
Something along these lines:

step(
        function f1() { find('foo',this); },
        function f2(err,foo) { if (foo) f4(foo); else return null; },
        function f3(err) { create('foo',this); },
        function f4(err,foo) { print(foo); }
);

If not, what is the best way to implement the code above?

Dynamic group tasks do not terminate if there are zero tasks

To put simply:

  var group = this.group();
  [].forEach(group());

And:

  var group = this.group();
  [].forEach(function() {
    doSomething(group());
  });

will never terminate.

I know this is by design, but maybe there should be some documentation of how to handle such situations?
Or possibly have dummy callbacks?

Node 0.4.0

Is this suppose to run on Node 0.4.0? I can't seem to get second function to work. It doesn't even seem to run it because first line after standard if(err) throw err; is sys.puts('second') and it doesn't output anything.

test.js just spits out lot's of stuff and I'm not sure how to determine if the test is passing or not.

Step is broken for use with Express.js!

Hi,

I started to create an app with the most popular node web framework Express.js and I discovered that Step might be broken if you want to use it inside of Express.

For me it seems that res.send() is causing Step to "step forward" which is a bad, bad thing. Look at the following example app:

var express = require("express");
var step    = require("step");

var app = express.createServer();

app.get("/", function(req, res)
{
    step(
        function one()
        {
            console.log("one");

            if (true)
                return res.send("one");

            this();
        },
        function two()
        {
            console.log("two");

            res.send("two");
        }
    );
});

app.listen(8080, function() {
    console.log("App is running on port 8080");
});

If you run this example and go in your browser to http://localhost:8080 this is what you get:

As you can see in the code, the output "two" actually never should be happen because the first function inside of step should always return.

IMHO this is a really crucial bug because it makes Step (which is awesome by the way) almost unusable for most node projects.

Cancel a Step Sequence

Is there a way of cancelling the step sequence? A "happy day" scenario for example might look like:

Serially, do A
Parallel, do [B,C]
Parallel, do [D,E]
Final callback F.

If an error happens in A, it appears you need to propagate it to F. Instead, I may prefer to just cancel all of the remaining Steps and execute a common error handling routine outside of my step sequence.

Race condition in handling of group()-generated callbacks

Hello

I believe there is a race condition in handling of group()-generated callbacks in Step. The problem is described in:

http://code.google.com/p/marmalade/issues/detail?id=35

which has been worked around by the following change in Marmalade:

http://code.google.com/r/gleberp-marmalade/source/detail?r=e61d87b61000ba7eaecebdd6482d52057e4e6227

Hope this helps!
Gleb Peregud

A way to step back to previous

I'd like a way to be able to set the step back however many times I need.

For example, setting it back 1 will execute the last action, then the action after that..

Basically, a way to alter the internal counter would be fantastic. :) (but functions like previous could also be helpful)

Are Closures supported? (closed accidentally)

I have this:

db.queryOne(query,insertCurso(curso,insertNotas(idAlumno,curso.id,notas,finRegistro)));

with insertCurso and insertNotas being closures, how can I implement this module in this situation?

step ignores wrong javascript code silently

When I execute the following program I expect it to crash or trow errors. However the first function is ignored and 'step 2' is printed on the console.

var step = require ('step');

if (!module.parent)
  step (
    function () {
      asdf // <=== THIS IS WRONG
      console.log ('step 1');
      this ();
    },
    function () {
      console.log ('step 2');
      this ();
    }
  )

Critical: Step is skipping steps!

Still trying to track down the exact issue, but here's a code snip:

getFileList( dir, dir, cb );
function getFileList( dir, root, cb ){
    var files
      , stats
      , outFiles = [];

    step( function(){
        fs.readdir( dir, this );

    }, function( err, _files ){
        if( err ){ throw err; }

        files = _files;

        var stats = this.group();
        for( var i=0, l=files.length; i<l; i++ ){
            fs.stat( path.join( dir, files[i] ), stats() );
        };

    }, function( err, _stats ){
console.log( 'HERE1' );
console.trace();
    }, function( err, files ){
console.log( 'HERE2' );
console.trace();
        cb( null, files );
    } );
}

from which I get (75% of the time):

HERE1
Trace: 
    at Function.<anonymous> (../docgen/fileList.js:42:9)
    at next (../docgen/node_modules/step/lib/step.js:51:23)
    at check (../docgen/node_modules/step/lib/step.js:73:14)
    at ../docgen/node_modules/step/lib/step.js:86:20
    at check (../docgen/node_modules/step/lib/step.js:101:9)
    at ../docgen/node_modules/step/lib/step.js:118:22
HERE2
Trace: 
    at Function.<anonymous> (../docgen/fileList.js:45:9)
    at next (../docgen/node_modules/step/lib/step.js:51:23)
    at Array.check [as 0] (../docgen/node_modules/step/lib/step.js:73:14)
    at EventEmitter._tickCallback (node.js:126:26)

The other 25% of the time it hangs as expected.

There needs to be a this.destroy

There has to be a this.destroy() that releases all the functions in a step, otherwise there is a memory leak in the case of any error or in the case that a function wants to exit without completing the steps.

Simple leak case:

step(function(){ return; }, function(){ });

2nd function is leaked. The 1st should be allowed to call this.destroy() before returning.

Remove nextTick check for groups.

Removing this line would be nice. :)
https://github.com/creationix/step/blob/master/lib/step.js#L102

This would cause issues when no group() has been called, but it would allow groups to be setup later (in callbacks), which would be pretty cool.

Change log

This project really needs a change log.

See http://keepachangelog.com

Problem if the first argument of a callback is not error

I recently tried using Step with path.exists, which doesn't pass an error argument as the first argument to the callback. Here's the script I tested:

var Step = require('step'),
    path = require('path');
var stuff = ['file1', 'file2', 'file3', 'file4'];
Step(
    function () {
        var group = this.group();
        stuff.forEach(function (item) {
            path.exists(item, group());
        });
    },
    function (err, list) {
        console.log(err);
        console.log(list);
    }
);

This should output null (or some other falsy value) for error, and an array containing false: [false, false, false, false]. Instead, we get:

$ node stepissue.js 
undefined
[ undefined, undefined, undefined, undefined ]

I don't think this is a version compatibility issue, but for good measure, I'm running Step 0.0.4, node 0.4.1 on Ubuntu 10.10.

process.nextTick(check) in next.group could cause a bug of race condition

I really like the way step works, simplifying my code greatly. However when I read the code of step, I'm really skeptical about this:

function check() {
  if (pending === 0) {
    // When group is done, call the callback
    localCallback(error, result);
  }
}
process.nextTick(check2); // Ensures that check is called at least once

I believe the comment "Ensures that check is called at least once" actually means:

If var group = this.group() is executed but no subsequent group() is called, we have to make sure the next step is called back anyway (with result equal to []).

That's right. But what if there are subsequent group() calls? If this next-tick-check is done before any asynchronous callbacks (returned by subsequent group() calls) are called, it's harmless, because pending can't be equal to 0. However, if the next-tick-check is called last, it could be disastrous because localCallback will consume another step function with the same result.

The thing is, you can't assume process.nextTick callback will be called in the very first place. This thread talks about it: https://groups.google.com/forum/?fromgroups=#!topic/nodejs/e8nLG5xrorA . From time to time process.nextTick callback really happens as the last one.

I have managed to construct a counter-example. Firstly, to facilitate watching, I changed step.js a little:

function check() {
  console.trace('check by callback');
  if (pending === 0) {
    console.trace('do by callback');
    // When group is done, call the callback
    localCallback(error, result);
  }
}
function check2() {
  console.trace('check by nextTick');
  if (pending === 0) {
    console.trace('do by nextTick');
    // When group is done, call the callback
    localCallback(error, result);
  }
}
process.nextTick(check2); // Ensures that check is called at least once

Then I run this test code:

var fs = require('fs'),
    path = require('path'),
    step = require('step');

var root = path.join(__dirname, 'test'),
    files;

step(
  function () {
    fs.readdir(root, this);
  },
  function (err, _files) {
    if (err) { throw err; return; }
    files = _files;
    var group = this.group();
    files.forEach(function (file) { fs.stat(path.join(root, file), group()); });
  },
  function (err, stats) {
    if (err) { throw err; return; }
    var group = this.group();
    files.filter(function (file, i) {
      return stats[i].isFile();
    }).forEach(function (file) { fs.readFile(path.join(__dirname, 'test', file), group()); });
  },
  function (err, contents) {
    if (err) { throw err; return; }
    console.log(contents);
  }
);

When there are only one file and one directory under test directory, I get the result, unfortunately, like this:

$ node test.js 
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at Object.oncomplete (fs.js:297:15)
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at Object.oncomplete (fs.js:297:15)
Trace: do by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:100:17)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at Object.oncomplete (fs.js:297:15)
Trace: check by nextTick
    at check2 (/home/djkings/mdblog/node_modules/step/lib/step.js:106:15)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)
Trace: do by nextTick
    at check2 (/home/djkings/mdblog/node_modules/step/lib/step.js:108:17)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)
[ { dev: 2051,
    ino: 4335031,
    mode: 33188,
    nlink: 1,
    uid: 1000,
    gid: 1000,
    rdev: 0,
    size: 1557,
    blksize: 4096,
    blocks: 8,
    atime: Tue Oct 16 2012 11:09:49 GMT+1300 (NZDT),
    mtime: Tue Oct 16 2012 10:49:08 GMT+1300 (NZDT),
    ctime: Tue Oct 16 2012 10:49:08 GMT+1300 (NZDT) },
  { dev: 2051,
    ino: 4335032,
    mode: 16893,
    nlink: 2,
    uid: 1000,
    gid: 1000,
    rdev: 0,
    size: 4096,
    blksize: 4096,
    blocks: 8,
    atime: Mon Oct 15 2012 17:48:52 GMT+1300 (NZDT),
    mtime: Sun Oct 14 2012 09:46:50 GMT+1300 (NZDT),
    ctime: Tue Oct 16 2012 10:49:16 GMT+1300 (NZDT) } ]
Trace: check by nextTick
    at check2 (/home/djkings/mdblog/node_modules/step/lib/step.js:106:15)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at fs.readFile (fs.js:176:14)
    at Object.oncomplete (fs.js:297:15)
Trace: do by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:100:17)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at fs.readFile (fs.js:176:14)
    at Object.oncomplete (fs.js:297:15)

However, when I add one more file to the test directory, the timing changes accordingly:

$ node test.js 
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at Object.oncomplete (fs.js:297:15)
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at Object.oncomplete (fs.js:297:15)
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at Object.oncomplete (fs.js:297:15)
Trace: do by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:100:17)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at Object.oncomplete (fs.js:297:15)
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at fs.readFile (fs.js:176:14)
    at Object.oncomplete (fs.js:297:15)
Trace: check by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:98:15)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at fs.readFile (fs.js:176:14)
    at Object.oncomplete (fs.js:297:15)
Trace: do by callback
    at check (/home/djkings/mdblog/node_modules/step/lib/step.js:100:17)
    at next.group (/home/djkings/mdblog/node_modules/step/lib/step.js:127:22)
    at fs.readFile (fs.js:176:14)
    at Object.oncomplete (fs.js:297:15)
[ <Buffer 23 20 56 6f 64 6b 61 0a 0a 23 23 20 4e 65 6d 69 72 6f 66 66 20 44 65 6c 69 6b 61 74 0a 0a 21 5b 4e 65 6d 69 72 6f 66 66 20 44 65 6c 69 6b 61 74 5d 28 76 ...>,
  <Buffer 23 20 47 69 6e 0a 0a 2e 2e 2e 74 68 65 20 62 65 73 74 20 64 72 69 6e 6b 20 69 6e 20 65 78 69 73 74 65 6e 63 65 20 69 73 20 74 68 65 20 50 61 6e 20 47 61 ...> ]
Trace: check by nextTick
    at check2 (/home/djkings/mdblog/node_modules/step/lib/step.js:106:15)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)
Trace: do by nextTick
    at check2 (/home/djkings/mdblog/node_modules/step/lib/step.js:108:17)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)
Trace: check by nextTick
    at check2 (/home/djkings/mdblog/node_modules/step/lib/step.js:106:15)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)
Trace: do by nextTick
    at check2 (/home/djkings/mdblog/node_modules/step/lib/step.js:108:17)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

I guess now you know what I mean. Ideally we should handle the empty group case just like handling the empty parallel case, checking it synchronously rather than by process.nextTick. As an exercise I'll try to fix this problem but it might involve major revision in step.js.

Can I preserve a callback across multiple steps?

My first step is a two db queries in parallel (this.parallel)

My second step is to do something with the data from one of those queries, but not the other, which I want to preserve for the next step. I don't want to group them because they are not the same.

My third step is to render them both.

qv:

step(
    function step1(){
      person.findById(req.params.person, this.parallel());
      blurb.find({ref:req.params.person}, this.parallel());
    },
    function step2(err, person, blurbs){
    // refine blurb object, preserve person, and send them both to the next step
    },
    function step3(err, person, blurbs){
      if (err){console.log(err);}
      console.log(person);
      console.log(blurbs);
    });

Missing instrunctions on how to run the tests

Ideally also provide a simple Makefile with a "check" rule, following GNU standards