bigeasy / proof Goto Github PK

Suggest using proof._coffee as the harness name instead of harness._coffee. Of course, a harness can be regular CoffeeScript, JavaScript or Streamlined JavaScript. Suggest test.t as a test name convention, because syntax highlighting can be discovered through the shebang line on most rational editors. Bring the examples in proof into line with the conventions.

Leading white space is stripped.

Currently, it might be possible for a callback be called back immediately, causing the next step execute immediately. We need to wait until the step function itself completes. There may be case where multiple callbacks are being requested, but the first one returns immediately so that the next step executes while the previous step continues to register callbacks.

Give some consideration to the difference between an exception and an error of usage. As it stands, it is not pretty to barf with a stack trace when the user specifies a test program twice in a test run. A simple error message plus a usage message ought to be enough.

Maybe error messages can go to: http://bigeasy.github.com/proof/errors#doubles where you can find more about this message.

When the PROOF_NO_CLEANUP environment variable is set, tests hang at exit.

I'm getting more enjoyment out of Makefiles.

It's nice to have the decimal points line up. If you know you're going to have a long running test, you can run your Proof test glob with a wider column for test run time.

Not detecting non-zero exists in proof errors unless there is also a failed assertion within the test.

Add the process id of test to the amalgamated asynchronous inquiry output, so we can report what might have been failure notes, such as exceptions. We don't want to get into the habit of parsing output. That falls into the trap of complexity.

Perhaps, if we're running at Travis CI, or running against the progress runner, we can emit JSON instead of using util.inspect. Maybe we just use JSON because it can parse.

Deleted the expected versus actual check when I implemented the asynchronous flows.

Set the width of the progress output.

Make asynchronous flow generic for use in cleanup and tests without harnesses.

Create proper usage strings for all of the sub-commands.

I use die as my general purpose freak-out function, so I'd like to use it here as well.

Implement cleanup according to my README.md where I claimed that it ran before and after each test. Currently, there is no such thing. I believe I have a start on some form of tear down, but I'm not doing teardown.

The runner output parser ought to be unforgiving and exit with an error message and non-zero exit code if an invalid runner output line is detected.

Failed assertions repeat the message name.

not ok 3 truthful # truthful

This was introduced by the throws assertion commit.

I've yet to test that exceptions are correctly thrown. It can be tedious to write a try/catch block that also needs to test that the exception was not thrown. Supposed you simply put a fail underneath the point in the block where the exception is thrown, but that doesn't fit with the logic of Proof. How do you tie the fail call to the equality test for the expected exception?

Do you have a method that returns an exception? It would be nice to be able to do a detailed inspection of an exception that has properties other than a message.

Because the context is a place for the user to sore things, we need to hide our private members with more than a single underscore. Already it collides with the Streamline.js callback. Need to add a prefix like __proof_, to go all out with the hiding.

I'd do this as part of the async commit, but it is going to be a noisy diff, so I'd like to isolate it in a single commit.

I'd rather be a JavaScript programmer, actually.

Remove the last of the README.md stuff that you've kept around in case you needed inspiration. Remove any petulance, defensiveness. Add syntax highlighting for JSON and CoffeeScript examples.

proof errors needs to exit with a non-zero exit code when errors are detected.

Maybe I want something simple like, UNTIDY.

When running more than one process using the default runner, multiple summaries are generated.

Some tests fail due to compilation errors. They fail immediately and are marked as successful runs, but with an undefined number of tests. They need to be marked as failures and the entire test run needs to be marked as a failure.

To reproduce, create a test program that will not compile.

A passed test when no tests are expected is reported as success. It should be reported as a failure. Tests are failures when their expected assertions do not match their actual assertions. I might be only checking to see if expected matches passed. Need have all three be equal.

Discontinue support of creating a context by assigning a object.

Create a flow for cleanup similar to the flow created to support JavaScript tests.

Implement clean up using a named function to indicate that the function is a cleanup function.

The proof errors command should exit with a non-zero exit code when errors are detected.

Organize tests by language, either node, coffee, or streamline.

We already have the ability to pluck a function bound to the Test object out of the Test object. Let's think of a name for generating a callback. What about this.stash or this.store or this.cubbyhole?

Remove it. While you're at it, remove the rambling motivations at the bottom of the README.md.

This is a euphemism. What we're really talking about here is creating an interface to bash to do the rigamarole of testing. Creating an extensible mechanism, like the one in git that allows users to extend the proof executable to suit the needs of their testing environment. That way, proof can continue to stay simple, providing the runner and some formatters. Pipelines in Node.js are not as easy as they are in bash, nor are they as easy as Node.js people say they are.

This makes proof less conducive to Windows. I'm okay with that. Windows is not a development platform.

Actually, it changes nothing, because piping was always part of Proof. This will only make the pipelines reusable.

Replace the current option parser with a minimalist option parser.

Need to run a spell checker against the README.md and the code comments.

Appears to only be showing the assertion success ratio of the last test run.

Currently, if a test hangs before the first assertion, there is no indication of which test has hung. The progress runner will have received a plan, but will not print to the console until it gets its first assertion. It should print a summary when it gets the plan.

Forgot about bail outs when implementing proof errors.

@say should accept a variable number of arguments like console.log.

When there are not enough digits to display a timing, the timing should expand the width of the timing display for the current run.

Gather up standard error and standard out prior to printing the plan. If the plan never comes and the test program instead exists with a non-zero value, then display the gathered output. This is the nature of test programs that failed to compile.

Create a single callback function that does it all. It stores, returns an error only callback, or acts as a callback.

This could be way too verbose for a long running test like the ones in TImezone, but for a handful of failed tests at Travis CI, this is going to be the shortest path to a working build.

If a test is not okay, all output that is not test runner output is printed until the next test runner output.

When a test fails, we only see what came after the failed test, so it if we add debugging output to see what lead up to the test, the proof errors display won't show it. This is a problem when trying to figure out what's going on remotely, like at Travis CI.

Support JavaScript tests with optional positional parameters. The problems with converting from CoffeeScript to JavaScript are the lack of a this short cut, so that we do not have a shorthand to get to our assertion functions, plus no destructured assignment, so we cannot simply unpack our context.

To deal with the assertions, I'm going to grab the $ variable and use that for Proof state. The $ variable is passed in as the first argument, if we detect it in the function signature. That acts as this. In fact, it can simply be a synonym for this.

Yes, I said detected in the function signature. Everything after $ and before the end of the function signature, or a _ is considered a parameter to unpack.

They seam rather shallow when testing Strata. It should be the case that in tests, the depth of of a deepEqual is not too deep, otherwise you can't see what's going on.

When running at the command line, it is probably enough to know that the test failed. You can rerun it directly. When running from Travis CI, you're going to probably want to know which assertions failed, especially if tests have been passing on your development machine.

Because we're not writing directly to a console, assertions are no longer animating.

Update the getopt submodule to 0.0.1. The getopt program is starting to use pre-release version numbers.