This isn't a concern as far as Mithril or ospec itself are concerned, but a test suite that make async requests may see its run time dominated by network and remote work latency. In these scenarios, it could make sense to have streaks (a task and its corresponding beforeEach and afterEach hooks) run in parallel.

In that scenario, having them all share a context object would make a lot of sense.

The v5 API has a nice affordance for this with the injected o object.

We could have o("some test", (o, {spy, context, timeout} = o) => {...}) or even o("", ({o, spy, context, timeout}) => {...}) (if o.o ===o).

context could prototypically inherit from the spec context passed to before and after hooks (each spec having its own context that inherits from parent specs).

A context object is necessary in parallel mode because we can't rely on the closure variables not to be clobbered by other streaks running in parallel.

We could use o.parallel(maxParallelism?, callback), or overload o.spec(name, options = {parallel: false|number}, callback).

The latter is probably preferable.

For sets, this does an object identity check for membership, I suppose we could do N^2 deepEqual() checks on the set members to determine if they match... Likewise for maps...

      if (a instanceof Set && b instanceof Set) {
        for (const el of a) {
          if (!b.has(el)) return false
        }
        for (const el of b) {
          if (!a.has(el)) return false
        }
        return true
      }

It is currently possible to have assertions that run after the test that defined them, and at that point, ospec is unacle to trace their origin.

One possibility to solve this would be to change the API to something tape-like:

o("test", o => {
  // this would either refuse to work
  setTimeout(()=>o(1).equals(false), 5)
})

If we were to implement this, we'd need a complimentary codemod to upgrade the test suites. If someone skilled in theses want to step up and write it would be most welcome. Otherwise I'll dive into it.

deepEquals always fails on objects with arrays created with seamless-immutable, (or any other library that adds methods to the arrays)
ospec version: 4.1.1

Code:

The following test fails when it shouldn't

let a = Immutable({
    tasks: {
        tasksList: [
            {
                due_date: 1595808000,
                dueMessage: 'Due today',
                daysLeft: 0,
            }
        ],
        computed: false
    }
})
, b = Immutable({
    tasks: {
        tasksList: [
            {
                due_date: 1595808000,
                dueMessage: 'Due today',
                daysLeft: 0,
            }
        ],
        computed: false
    }
})

o('test', () => {
    o(a).deepEquals(b)
})

o.run()

The problem seems to be here: https://github.com/MithrilJS/ospec/blob/master/ospec.js#L543

Ospec is using Object.getOwnPropertyNames on the array so it's finding non-enumerable functions and trying to compare them as well, and so it's finding non-enumerable functions and trying to compare them as well, rather than just ignoring them like it should.

In the Mithril test suite, @dead-claudia uses a lock utility that tracks stray calls to functions after a test is finished. This should be a core functionality of o.spy()

The groundwork I'm doing to fix #50 should make the default case easy (spies defined during a test should not run when it's done. Spies defined at test-definition time shouldn't expire though.

It may be possible to have spies defined in before/beforeEach hooks to last until the matching after/afterEach hook.

Somehow, I missed this entirely when running the tests.

Mithril version:

Browser and OS:

Project:

Is this something you're interested in implementing yourself?

Description

ospec needs integrated with Karma so ospec users can use Karma for much easier cross-browser testing.

Why

This is mildly blocking for us moving to Karma internally - the only alternative is literally just moving to another framework. Also, users may appreciate this.

Possible Implementation

Open Questions

Platform: Windows 10, 64 bit
Node: v10.16.0
NPM: 6.9.0

I'm trying to reproduce a testing setup from official docs.
When I run npm test either from powershell or cmd I get an error:

> ospec --require ./test-setup.js

internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module './test-setup.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.resolve (internal/modules/cjs/helpers.js:33:19)
    at C:\Users\Predator\Desktop\dev\JS\mithril repos\test-test\node_modules\ospec\bin\ospec:33:31
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (C:\Users\Predator\Desktop\dev\JS\mithril repos\test-test\node_modules\ospec\bin\ospec:31:15)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
npm ERR! Test failed.  See above for more details.

Then I change a line in package.json from
"test": "ospec --require ./test-setup.js"
to
"test": "ospec --require ../../../test-setup.js"

And I get another error caused by require("mithril") line in test-setup.js (there was no mention to install mithrill in the docs)

Then I do npm install --save-dev mithril

And I get another error

Error: Cannot find module '../../../test-setup.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.resolve (internal/modules/cjs/helpers.js:33:19)
    at C:\Users\Predator\Desktop\dev\JS\mithril repos\test-test\node_modules\mithril\ospec\bin\ospec:33:31
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (C:\Users\Predator\Desktop\dev\JS\mithril repos\test-test\node_modules\mithril\ospec\bin\ospec:31:15)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3

Then I change a line in package.json from
"test": "ospec --require ../../../test-setup.js"
to
"test": "ospec --require ../../../../test-setup.js"

And I get output:

> ospec --require ../../../../test-setup.js

––––––
All 0 assertions passed in 5ms

However expected output according to docs is:

––––––
All 1 assertions passed in 0ms

As discussed in gitter: https://gitter.im/mithriljs/mithril.js?at=5d6b2250418f705d876fc8d6

@isiahmeadows felt this would be low-effort to support.

Is this something you're interested in implementing yourself?

No, sorry, not quite my expertise 😬

I love ospec and its simplicity. Two questions:

1. I didn't see this documented, but is it possible to view the test progress in real time. For example, many test runners will show a stream of . or X as each test passes or fails, and these are written to the terminal as soon as the result as in, rather than waiting until all test results are in, and showing them all at once.
2. If not possible, are there are any plans for this? If not, would this be a welcome feature PR? And if so, how difficult would retrofitting such a feature be?

The report currently leaves out the own properties of errors caught by the runner. I would be nice to have them show up as it happens when they are thrown or logged normally.

Suppose I have a test like this:

o('%foo').equals('%ioo')("%doo")

When evaluated, I'd expect for this expression to produce the following output:

%doo

'%foo'
  should equal
'%ioo'

What I'm getting instead is this:

0oo

'NaNoo'
  should equal
'NaNoo'

Now I'm guessing this has to do with format placeholders, but I'm fairly sure that any such substitution should only be performed once (so that it does not affect the inserted text).

Edit: checked o.spec("%foo", () => o("%foo", () => {/* … */})) – these strings are also affected.

Platform: Windows 10, 64 bit
Node: v14.17.0
NPM: 6.14.13
ospec: 4.1.1

Reproduction Steps

1. npm install --save-dev ospec
2. Change test command in package.json to ospec
3. Add tests/mytest.spec.js with following:

const o = require('ospec');
const assert = require('assert');

o.spec("my test", () => {
    o("things are working", () => {
        assert.strictEqual("foo", "bar");
    });
});

Expected

Just a message saying 0 assertions passed

Actual

0 assertions passed, with 1 assertion bailed out with following: (note: I altered the name of the project directory in output)

Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only file and data URLs are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:782:11)
    at Loader.resolve (internal/modules/esm/loader.js:88:40)
    at Loader.getModuleJob (internal/modules/esm/loader.js:241:28)
    at Loader.import (internal/modules/esm/loader.js:176:28)
    at importModuleDynamically (internal/modules/cjs/loader.js:1011:27)
    at exports.importModuleDynamicallyCallback (internal/process/esm_loader.js:30:14)
    at eval (eval at <anonymous> (C:\Users\JamesCote\Projects\my-project\node_modules\ospec\bin\ospec:24:15), <anonymous>:1:1)
    at C:\Users\JamesCote\Projects\my-project\node_modules\ospec\bin\ospec:24:15
    at C:\Users\JamesCote\Projects\my-project\node_modules\ospec\bin\ospec:75:15
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  code: 'ERR_UNSUPPORTED_ESM_URL_SCHEME'
}

C:\Users\JamesCote\Projects\my-project\tests\mytest.spec.js > > > BAILED OUT < < <:
Only file and data URLs are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only file and data URLs are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:782:11)
    at Loader.resolve (internal/modules/esm/loader.js:88:40)
    at Loader.getModuleJob (internal/modules/esm/loader.js:241:28)
    at Loader.import (internal/modules/esm/loader.js:176:28)
    at importModuleDynamically (internal/modules/cjs/loader.js:1011:27)
    at exports.importModuleDynamicallyCallback (internal/process/esm_loader.js:30:14)
    at eval (eval at <anonymous> (C:\Users\JamesCote\Projects\my-project\node_modules\ospec\bin\ospec:24:15), <anonymous>:1:1)
    at C:\Users\JamesCote\Projects\my-project\node_modules\ospec\bin\ospec:24:15
    at C:\Users\JamesCote\Projects\my-project\node_modules\ospec\bin\ospec:75:15
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

––––––
The 0 assertion passed (old style total: 1). Bailed out 1 time
npm ERR! Test failed.  See above for more details.

A possible workaround is to downgrade Node.js to v10.15.2, but perhaps this should be investigated to allow for support on newer versions as well.

o({ [Symbol("a")]: 1 }).deepEquals({}) // passes, but IMO shouldn't.

flems

The reporter should properly display error with causes and aggregate errors.

Not sure if it is still relevant, bu I have this flems with a fix for a problem with ImmutableJS that was in a stray tab. Keeping it here until I find out if it is relevant.

Currently, if a crash happens in the deepEquals code (as in #41), or in a .satisfies plugin, this is reported as a bailout.

Crashes in user code (e.g. an getter that throws when doing a deepEquals, or some other failure in user code called from the satisfies() validator) should cause bailouts, but errors in ospec and validators should throw errors and exit ASAP with a corresponding stack trace.

const o = require('ospec')

o.spec('Example', () => {
  o.before(() => {
    console.log("RUNNING BEFORE")
  })

  o('test1', () => {
    o(true).equals(true)
  })
})

o.only('test2', () => {
  o(true).equals(true)
})

In this example, the console log runs even though no test in that spec is running (due to the .only on test2). In my own project this is causing extra debug statements to show up in the console even though they're not relevant to the test I'm focusing on.

the done parser doesn't care about comments with carriage return

if we use git with "checkout Windows-style, commit Unix-style line endings" setting on Windows, the test "the done parser > tolerates comments" fails

maybe same issue MithrilJS/mithril.js#2351

The design would be conceptually simple: expose o.define = define where define(key, cond) is an internal function we already use for similar effect, and expand that function to take more than 1 argument.

To define a custom assertion, you'd call o.define("method", (operand, ...args) => cond). You would then call this via o(operand).method(...args). It's conceptually pretty simple, and it's easy to implement.

This would make things like mithril-query much easier to integrate, since that isn't as simple as equality or deep equality.

ospec version: 4.0.1

Browser and OS: Node v12.12.0 & Firefox v71

Code

o("addition", function() {
	o(1 + 1).equals(3)("addition should work");
});
o.run();

demo in flems

Steps to Reproduce

1. Run the above.

Expected Behavior

According to the documentation the assertion's message should override the test's description like so:

/* for a failing test, an assertion with a description outputs this:

addition should work

1 should equal 2

Error
  at stacktrace/goes/here.js:1:1
*/

Current Behavior

The assertions message is never displayed

Hi
Thanks a lot for making and supporting ospec.
I would like to ask you to provide ES6 version of ospec.
We are currently in the process of migrating to Rollup and we try to use native ES6 modules for development, in both web and node versions. To use ospec we use @rollup/plugin-commonjs which transforms commonjs module into ES6 module (or tries its best to do so). Unfortunately it cannot do much with dynamic require() for the util which is not allowed when the file is treated as ES module. What could be used for ES there is dynamic import().

I haven't dug yet, but I think we could integrate https://github.com/TypeStrong/ts-node/ and load it on demand if needed.

The devDependencies make no sense in npm package, just make the installation longer and heavier.

"devDependencies": {
        "compose-regexp": "0.4.0",
        "eslint": "^6.8.0",
        "ospec": "4.0.1"
}

If the sales point is light weight, above should be get rid off before publish.

ospecs messaging processes assertion values differently depending on environment. Under Node it uses util.inspect, the platforms preferred tool for debugging; but in the browser it falls back to a loosely opinionated custom process for stringifying values.

Browser consoles are much richer than Nodes' terminal output and could be better served by making use of string substitution, which allow intelligent representations of complex entities - see below for current vs suggested behaviour for exotic objects logged by ospec in Chrome

Happy to implement this myself.

ospec version: 4.1.1
node version: 14.15.4

throws does not seem to catch exceptions from async functions.

Code

o("synchronous", function() {
    o(() => {throw new Error()}).throws(Error)
})
o("asynchronous", function() {
    o(async () => {throw new Error()}).throws(Error)
})

Expected behavior

––––––
All 2 assertions passed (old style total: 2) 

Current behavior

asynchronous:
[AsyncFunction (anonymous)]
  should throw a
[Function: Error] { stackTraceLimit: 10 }
    at /home/vis/dev/repositories/2020-04-pqmail-prototype/report.js:6:40
    
––––––
1 out of 2 assertions failed (old style total: 2)  

o("x", ()=> {
  o(Object.create(null)}.deepEquals(Object.create(null))
})

This ends up at the last line of the deepEquals logic, and fail because neither have a .valueOf() method.

I'm working on a fix.

PhantomJS runner:

var o = require("./ospec.js")
require("./polyfill.min.js")
require("./tests/test-ospec.js")
o.run()

Polyfill: https://polyfill.io/v3/polyfill.min.js?flags=gated%2Calways&features=Promise

Output:

$ phantomjs test.js
%cospec > sync > spy wrapping:%c
%cAttempting to change value of a readonly property.%c
defineProperties@[native code]
 color:black;font-weight:bold  color:red color:black

  phantomjs://platform/ospec.js:356 in report
%c1 out of 371 assertions failed%c  in 359ms color:red;font-weight:bold

I'm not able to test components having import statement with it.

var mq = require("mithril-query");
var o = require("ospec");

import { Alert } from "[path]";

o.spec("Alert", function() {
  o("Alert", function() {
    var out = mq(Alert, { type: "danger" }, ["You got it"]);
    out.should.have("alert-danger");
    out.should.contain("You got it");
  });
});

o.run();

I'm getting error as below

import {Alert} from "[path]"
^^^^^^

SyntaxError: Cannot use import statement outside a module

We currently shove every kind of problems into the results array as assertions, even timeouts and bail outs.

The latter two should be tagged as such IMO.

Also, we shouldn't format the output on error, but in the reporter (with a possible exception for custom assertions).

Here's the current format:

{
	pass: null,
	message: "Incomplete assertion in the test definition starting at...",
	error: $$_testOrHook.error,
	task: $$_testOrHook,
	timeoutLimbo: $$_timedOutAndPendingResolution === 0,
	// Deprecated
	context: ($$_timedOutAndPendingResolution === 0 ? "" : "??? ") + $$_testOrHook.context,
	testError: $$_testOrHook.error
}

We should aim for something like this:

{
	kind: "assertion" | "error" | "timeout",
        pass: boolean,
	message?: {value, methodName, reference} | string
        error?: Error // or maybe just a stack trace?
	task: Task
}

This is a prerequisite for #31.

The "incomplete assertion" checks should run on task finalization time, and on timeout, and cause a hard error to be thrown.

Likewise, syntax errors while parsing a test file should cause hard errors, not bailouts.

At long last, it would be useful to be able to bulk-add results, such that we can parallelize running the tests and merge results.

Runtime looks like this and works fine:

const res = {
  end: o.spy()
}
await handler(req, res)
console.log("First call", res.end.calls[0])
//=> First call { this: <ref *1> { [omitted]  }, args: [ 'ok' ] }

However, TS complains about further usage:

It thinks .calls is an array of arrays:

In o(myObject).satisfies(myValidator(someSepcification)), the satisfies bit takes too much space and hampers readability. o(myObject)._(matches(someSepcification)) would make more readable tests.

Also, the .notSatisfies() and .notDeepEquals() assertions are not very useful, and even IMO an anti-pattern, as any difference will make them pass, not necessarily the one the tester had in mind when designing the test. They should be. retired.

Here's the roadmap I propose

For v4.2.0:

Add ._() as a synonym to .satisfies().

Make o.warn = false. If the user sets o.warn = true, have .satisfies, .notSatisfies and .notDeepEquals issue warnings announcing future obsolescence (we can do that cleanly such that a single list is reported after publishing the report).

For v5.0.0

Make the warning unconditional, and state that the methods are deprecated.

Sometimes in the future

Remove them altogether.

ospec: Asynchronous tests doesn't work with arrow functions

ospec version: 4.0.1

Browser and OS: MacOS

Code

Following test works:

o("setTimeout calls callback", function(done) {
	setTimeout(done, 10)
})

But this test with arrow function gives and error:

o("setTimeout calls callback", (done) => {
	setTimeout(done, 10)
})

Steps to Reproduce

1. Create a test.js file with following:

var o = require("ospec")

o("setTimeout calls callback", (done) => {
	setTimeout(done, 10)
})

2. Run ospec test.js

Expected Behavior

––––––
The 1 assertion passed in 23ms

Current Behavior

ospec.js:181
						else throw e
						     ^

`(done)()` should be called at least once
    at Object.<anonymous> (/Users/kesara/Lab/ospec-test/test.js:3:1)

Context

ospec should treat the arrow functions as same as regular functions.

As seen here

  o(true).equals(false)(`Some ${"interpolated"} text`)

works, but

  o(true).equals(false)`Some ${"interpolated"} text`

doesn't.

We advertise the tagged template syntax in the docs (without interpolations, but still, this is tempting).

As part of an internal log an report upheaval that would also fix #31, I'd like to explicitly mark bail out, and split the report between normal failures at the top, then a report on test success (for those that passed) followed by a bailout report, if any.

This should be pretty straightforward, and it'd let me add PhantomJS to our test matrix.