Hi!

I have a qunit testsuite for a complete site. How do I (if ever possible) run it under testling.com?

TIA,
--Vladimir

Over at TraceKit.js we're trying to get some unit testing going: occ/TraceKit#11 (comment)

The key comment from @airportyh there:

Alright, so I had a go with testling and had high hopes for it, because it is really cool how all it requires is for your test framework to output the tap protocol via console.log, so I used Jasmine with the tap reporter. However, ultimately it didn't work for TraceKit because it bundles all of the source code into a string to be executed via eval, which kills stacktraces for TraceKit. I tried instead to hotlink to github for all the source files, but only to realize we are now block by the single-origin policy.

Was tape-testing/tape#31

Moving the issue to testling because the tests run without issue locally in FF 18 so the issue seems to be with testling or testling+tape

Thoughts?

I think there's some terrible cacheing going on with ci.testling.com. I've been having a really hard time getting https://github.com/devinrhode2/highlight-bubble to work with testing at all. I forked substack/defined as a starting point for a working example. I've been making the 2 repos more and more similar (copying pasting over files and code) and finally took the files of devinrhode2/highlight-bubble and just pasted them entirely into devinrhode2/defined, committed it with the wrong git url's and everything in package.json - IT STILL WORKS with the wrong git url's etc. Right now devinrhode2/highlight-bubble and devinrhode2/defined are identical repos, but only /defined works with testing.

BLARG

nested tests do not work in node or in the browsers.

Nested tests work in node-tap so it must be a bug in your harness code. (since it worked in node in testling before the harness refactor)

I have node v0.10.11 installed, and testling (and browserify) installed globally.

When I run testling in a directory with a properly formatted package.json, I get the following output:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: spawn ENOENT
    at errnoException (child_process.js:980:11)
    at Process.ChildProcess._handle.onexit (child_process.js:771:34)

What information can I provide to help fix this?

I cannot use the simple example from testling.com using createWindow:

var test = require('testling');
test('the test name', function (t) {
    t.equal(2 + 2, 4);

    t.log(window.navigator.appName);

    t.createWindow('http://substack.net', { t : t }, function (win, $) {
        t.equal(win.document.title, 'The Universe of Discord');
        t.end();
    });
});

I do get the t.log output but then, the createWindow hangs indefinitely.

Hi James,

While following the Testling Guide for Mocha: https://ci.testling.com/guide/mocha I attempted to verify my configuration locally using testling CLI and got the error:

not ok 1 Error: TypeError: 'tap' is not a constructor (evaluating 'new Reporter(runner)') on line 4129

Running npm test works fine.

Code: https://github.com/nelsonic/learn-testling

Any insight would be much appreciated.
Thanks.

sometimes you want test server to run on specific port and it will be great if it will be possible to add --port 5000 for example to run on 5000 port

HI, I try to user

sudo npm install -g testling.

I got the error

npm http 200 https://registry.npmjs.org/uglify-js
npm http GET https://registry.npmjs.org/uglify-js/-/uglify-js-1.1.1.tgz
npm http 200 https://registry.npmjs.org/uglify-js/-/uglify-js-1.1.1.tgz
/usr/local/bin/testling -> /usr/local/lib/node_modules/testling/bin/cmd.js

[email protected] install /usr/local/lib/node_modules/testling
browserify node_modules/schoolbus/proxy.js -o static/proxy.js

/usr/local/lib/node_modules/testling/node_modules/mkdirp/index.js:74
throw err0;
^
Error: EACCES, permission denied '/Users/feifeipan/.config/browserify'
at Object.fs.mkdirSync (fs.js:483:18)
at Function.sync (/usr/local/lib/node_modules/testling/node_modules/mkdirp/index.js:55:12)
at module.exports (/usr/local/lib/node_modules/testling/node_modules/browserify/node_modules/deputy/index.js:10:12)
at new Wrap (/usr/local/lib/node_modules/testling/node_modules/browserify/lib/wrap.js:32:30)
at module.exports (/usr/local/lib/node_modules/testling/node_modules/browserify/lib/wrap.js:17:12)
at module.exports (/usr/local/lib/node_modules/testling/node_modules/browserify/index.js:55:13)
at Object. (/usr/local/lib/node_modules/testling/node_modules/browserify/bin/cmd.js:87:14)
at Module._compile (module.js:449:26)
at Object.Module._extensions..js (module.js:467:10)
at Module.load (module.js:356:32)
npm ERR! [email protected] install: browserify node_modules/schoolbus/proxy.js -o static/proxy.js
npm ERR! sh "-c" "browserify node_modules/schoolbus/proxy.js -o static/proxy.js" failed with 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the testling package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! browserify node_modules/schoolbus/proxy.js -o static/proxy.js
npm ERR! You can get their info via:
npm ERR! npm owner ls testling
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 11.4.0
npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "testling"
npm ERR! cwd /users/feifeipan/Sites/myweb/node/node_modules
npm ERR! node -v v0.8.9
npm ERR! npm -v 1.1.61
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /users/feifeipan/Sites/myweb/node/node_modules/npm-debug.log
npm ERR! not ok code 0

Then I use "npm install testing", I got another error.

npm http 304 https://registry.npmjs.org/uglify-js
npm http 304 https://registry.npmjs.org/traverse
Checking for program g++ or c++ : not found
/usr/local/lib/node_modules/testling/node_modules/shoe/node_modules/sockjs/node_modules/rbytes/wscript:12: error: could not configure a cxx compiler!
npm WARN optional dep failed, continuing [email protected]
npm ERR! Error: EACCES, symlink '../lib/node_modules/testling/bin/cmd.js'
npm ERR! { [Error: EACCES, symlink '../lib/node_modules/testling/bin/cmd.js']
npm ERR! errno: 3,
npm ERR! code: 'EACCES',
npm ERR! path: '../lib/node_modules/testling/bin/cmd.js' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! System Darwin 11.4.0
npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "testling"
npm ERR! cwd /users/feifeipan/Sites/myweb/node/node_modules
npm ERR! node -v v0.8.9
npm ERR! npm -v 1.1.61
npm ERR! path ../lib/node_modules/testling/bin/cmd.js
npm ERR! code EACCES
npm ERR! errno 3
npm ERR! stack Error: EACCES, symlink '../lib/node_modules/testling/bin/cmd.js'
npm ERR! Error: EACCES, open 'npm-debug.log'
npm ERR! { [Error: EACCES, open 'npm-debug.log'] errno: 3, code: 'EACCES', path: 'npm-debug.log' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! System Darwin 11.4.0
npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "testling"
npm ERR! cwd /users/feifeipan/Sites/myweb/node/node_modules
npm ERR! node -v v0.8.9
npm ERR! npm -v 1.1.61
npm ERR! path npm-debug.log
npm ERR! code EACCES
npm ERR! errno 3
npm ERR! stack Error: EACCES, open 'npm-debug.log'
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /users/feifeipan/Sites/myweb/node/node_modules/npm-debug.log
npm ERR! not ok code 0

Is there anyone can help me?Thanks a lot.

I'm trying to set up CI on ForbesLindesay/queue as a simple project, before moving on to something more ambitious, but at the moment it just won't load anything for the results http://ci.testling.com/ForbesLindesay/queue

The preprocess command is not waited on before moving on to unglobbing, meaning that if the preprocess function is used to generate the test files (for example I use gulp to generate a single test bundle).

This means that the next step to unglob the files fails as it cannot find the script to be executed. Example package.json testling field

"testling": {
    "preprocess": "build-tests.bat",
    "scripts": "build/tests.bundle.js",
    "browsers": [
      "ie/8..latest",
      "chrome/latest",
      "firefox/latest"
    ]
  }:

To get chrome to launch locally I had to upgrade testling's browser-launcher version:

"browser-launcher" : "~0.1.0"

I'm running OSX 10.8.2 and Google Chrome 22.0.1229.94.

Can merge kesla/node-headless#10 and detect errors or do a precheck and run with a head where needed so this can support osx without a system requirement

When I insert a files key into package.json it works just fine using testling -u and then opening link in local browser:

But then fails at ci.testling:

So I have tests passing in testling locally but not on the ci server. This is bad. Why is the es5-shim missing from the ci version?

After installing the testling packages running npm I get the following error running the example test

$ testling test.js
The "sys" module is now called "util". It should have a similar interface.
node/jsdom                      0/0    0 % ok
test.js                         0/0    0 % ok
  Error: Cannot find module 'testling'
    at Function._resolveFilename (module.js:334:11)
    at Function._load (module.js:279:25)
    at Module.require (module.js:357:17)

Here is my npm version:

$ npm --version
1.0.104

My node.js version:

$ node --version
v0.6.1

I have mocha tests trying to load images:

img.src = '/test/fixtures/test.gif';

local mocha works because it perhaps starts a test server in the CWD. But ci.testling.com does not works because I do not know how to tell testling: copy '/test/fixtures/test.gif' to the build used by testling.

Should I just use files: ['/test/fixtures/*'] ?

and put all the browser/harness.js code into a seperate project that then just pipes TAP into browser/output.js

This way you can use any testing framework as long as it can output TAP and then pipe that into browser/output.js

Hi. Thanks for such a great project.

I'm trying to add testling into the https://github.com/bitpay/bitcore CI system. Our build process includes running node-gyp to compile some C++ code. This seems to be failing at ci.testling.com. The error we get can be found at https://gist.github.com/maraoz/10297571, but what I understood is that there is a problem with the python multiprocessing module. From what I read from Stack Overflow (http://stackoverflow.com/a/2009505/28802 and http://stackoverflow.com/a/3699231/28802), the fix is as simple as adding the following to your /etc/fstab:

none /dev/shm tmpfs rw,nosuid,nodev,noexec 0 0

If this fix is not possible, is there a way to make ci.testling.com run another build process than npm install?(which we might configure not to run node-gyp, which is not needed for the browser version (of course, browsers cannot run c++ bindings))

When running testling with a custom server field, the process does not die when the tests finish (regardless of whether they pass or fail). I am currently seeing this on OS X 10.9.2 with node v0.10.26. Anyone else seen this issue?

Unclear how to setup testing using the server aspects of testling. Some initial documentation or point to an example. I am also happy to create an example for the engine.io-client module and engine.io-stream module.

I've been trying to use the pkg.testling.html option detailed here, but I can't get it running no matter what I try.

The HTML example repo is a few months old, has this feature been deprecated, or am I missing something?

You can see the failed build here
https://ci.testling.com/helloblock/helloblock-js

In particular, I am having an error loading the main module:

Error: module "../lib/helloblock" not found from "/home/helloblock/repos/76a56a267d0f450e5c07d488ccf230616cd86039.1397335801064/test/addresses.spec.js"
* preprocessing command exit code 1 *

However, this code works perfectly locally:

https://github.com/HelloBlock/helloblock-js/blob/feature-testling/test/addresses.spec.js#L3

var mocha = require("mocha");
var expect = require("chai").expect;
var helloblock = require("../lib/helloblock")({
  network: 'testnet',
  debug: true
})

The module: ../lib/helloblock does exist, any idea why testling-ci is not able to find it?

ci.testling isn't using the latest version you have here,
which is causing tests to fail which should succeed.

because this isn't isn't updated:
https://github.com/substack/testling/blob/master/browser/prelude.js#L87-L91

Also, i noticed this:
https://github.com/substack/testling/blob/master/bin/cmd.js#L25

it tries to read from ../bundle/prelude.js but that directory is now called ../browser/prelude.js

After publishing my new package that uses testling ci
https://github.com/Gozala/interset

I have noticed only two browser logos on the github page while if I click the badge and enter
testling page I see all of them:
http://ci.testling.com/Gozala/interset

I thought it was some strange browser caching, but then discovered that on github image url is not
http://ci.testling.com/Gozala/interset.png as specified in readme but following instead:

https://a248.e.akamai.net/camo.github.com/e1e0a920e49f2c8d18a6536bd094cbd8f466b9be/687474703a2f2f63692e746573746c696e672e636f6d2f476f7a616c612f696e7465727365742e706e67

Presumably github caches and copies http content to avoid mixing it with http with https since that considered insecure. Either way this is nasty issue that would lie about browser compatibility.

P.S.: It also looks like caching is per data center or something since few other people sew diffirent pictures, although they all confirmed that they have diff img url from expected
http://ci.testling.com/Gozala/interset.png

Xvfb processes are left running after tests finish, and subsequent test runs hang until the processes are killed manually. I was able to get some success with a process.on('exit', function () {childProcess.kill()}) in headless, but ugh this Xvfb business is touch and go. This is on Arch Linux running testling 1.7.0.

I believe this issue is also mentioned in #71, though it is not constrained to custom servers.

Just curious if it will get in. Are you accepting PRs?

Latest Chrome Stable is v28 but testling is at v25
Latest Firefox is v23 but testling is at v19

Hi James,
following the Testling quick_start guide: https://ci.testling.com/guide/quick_start
I'm at the configuring the web hook stage and cannot find tesling in the "AVAILABLE SERVICE HOOKS" list.

see screenshot.

What am I missing?
(thanks!)

I reproduces this on a mac, where I updated browser-launcher to 0.2.0

To reproduce run this testing file with testling example/localhost.js --browser=chrome.

var test = require('../');

test('title test', function (t) {
    var w = t.createWindow('http://localhost:17000/', { t : t });
    w.next(function (win, $) {
        t.equal(win.location.href, 'http://localhost:17000/');
        t.equal(win.document.title, 'Localhost Test');
        t.end();
    });
});

The localhost server looks like this:

var http = require('http');
var path = require('path');
var filed = require('filed');

var server = http.createServer(function (req, res) {
    req.pipe(filed( path.resolve(__dirname, 'index.html') )).pipe(res);
});

server.listen(17000, '127.0.0.1', function () {
    console.log('ready on 127.0.0.1:17000');
});

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Localhost Test</title>
    </head>
    <body>
        Hallo World
    </body>
</html>

It should be noted that testling example/window.js --browser=chrome works perfectly!

Using the client-side testling module breaks if your server is not set up to handle the shoe / push thing.

For example a server I have returns a HTML string which shoe internally tries to JSON parse which falls over.

Alternatively have shoe try catch around the json parser or not fall over. (difficult to do as it is in some XHR abstraction callback in some dependency)

On node 0.10.0, testling list crashes immediately after running. This might be an issue with browser-launcher rather than testling, but I couldn't immediately tell.

> testling list
# detecting browsers for the first time

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: spawn ENOENT
    at errnoException (child_process.js:945:11)
    at Process.ChildProcess._handle.onexit (child_process.js:736:34)

I tried to run locally the following test using the command line testling test.js.

// Load testling dependency 
var test = require('testling');

// Load js files
require.load('./service.js');
require.load('./js/jquery.js');

test('service.createClient()', function (t) {
    var client = Service.createClient();
    t.ok(client);
});

and I got:

TypeError: Object function require(path) {
    return Module._load(path, self);
  } has no method 'load'
    at Object.<anonymous> (/home/lbdremy/workspace/nodejs/service/test.js:5:9)
    at Module._compile (module.js:402:26)
    at Object..js (module.js:408:10)

require.load doesn't exist. I used the example available at http://testling.com/docs/#require.load and I get the same error.
I installed testling from npm, the version published on npm is it up to date?

Could you help me, thanks.

git clone https://github.com/substack/testling.git
cd testling
npm update
node server

SyntaxError: Unexpected token }
    at Object.parse (native)
    at /home/dominic/dev/testling/node_modules/browserify/index.js:384:41
    at Array.reduce (native)

this is browserify crashing - it's actually from ~/dev/testling/node_modules/dnode/node_modules/socket.io/support/socket.io-client/lib/vendor/uglifyjs/package.json

maybe it would be better to eval the package.json?

It's not super-clear from the README, but it would seem that headless/non-phantom browser support for this module is Linux-only?

ci.testling is failing for 4b4b665987e66d36d4249dbffbfe5472a9d8b2cc in https://ci.testling.com/hughfdjackson/immutable-hash-trie on chrome 4. Debugging in browserling shows that the tests are passing as expected (see attached image).

It seems to be failing in the same way for all non-ie browsers, so I suspect it's not just an intermittent failure.

Couldn't figure whats wrong in https://ci.testling.com/pocesar/ES5-Class

When opening https://github.com/pocesar/ES5-Class/blob/master/test/index.html in the browser, it works. but not with Testling. Tried for hours to make work, but I couldn't. Since I'm on Windows, I also can't test locally running testling command. I also tried to check the bundle.js from browserify and couldn't find anything wrong with it, but it also failed to run on the browser, don't know what else to try.

I keep getting this error and I am not sure if others run into the same thing:

➜  helloblock-js git:(feature-testling) testling -u

Error: module "./lib-cov/mocha" not found from "/MY/PATH/node_modules/mocha/index.js"
FAILURE: non-zero exit code

I was writing some code for a few hours, running tests with Testling locally. I happened to glance at my activity monitor and noticed about 30 Phantomjs processes running.

I realized that the process didn't exit when I manually stopped a Testling test run.

Any idea how to prevent this? It kills my laptop after a couple hours.

To Reproduce:

• Test Web Service hook for testling ci on a known-good test run

Expected:

• The tests to succeed

Actual:

• tests error with "not ok 1 Error: Expected ';' on line 1"

Seen on https://github.com/hughfdjackson/persistent-hash-trie, https://github.com/hughfdjackson/immutable and https://github.com/ljharb/object-keys

testling crashes processing any code containing a bind call..

this test:

test("this", function(t){

    function sum(a, b){ return a + b; }

    t.equal(sum(1, 2), 3);

    var increment = sum.bind(null, 1);
    t.equal(increment(1), 2);

    t.end();
});

crashes with this output:

TAP version 13
# this
ok 1 should be equal
not ok 2 TypeError: 'undefined' is not a function (evaluating 'sum.bind(null, 1)')
  ---
    operator: error
    expected:
      undefined
    actual:
      { line: 7991, message: '\'undefined\' is not a function (evaluating \'sum.bind(null, 1)\')', sourceId: 4471247232, sourceURL: 'http://localhost:58830/__testling?show=true', stack: 'TypeError: \'undefined\' is not a function (evaluating \'sum.bind(null, 1)\')\n    at http://localhost:58830/__testling?show=true:7991\n    at http://localhost:58830/__testling?show=true:7134\n    at next (http://localhost:58830/__testling?show=true:6944)\n    at http://localhost:58830/__testling?show=true:9646', stackArray: [ { line: 7991, sourceURL: 'http://localhost:58830/__testling?show=true' }, { line: 7134, sourceURL: 'http://localhost:58830/__testling?show=true' }, { function: 'next', line: 6944, sourceURL: 'http://localhost:58830/__testling?show=true' }, { line: 9646, sourceURL: 'http://localhost:58830/__testling?show=true' } ] }
    stack:
      TypeError: 'undefined' is not a function (evaluating 'sum.bind(null, 1)')
        at http://localhost:58830/__testling?show=true:7991
        at http://localhost:58830/__testling?show=true:7134
        at next (http://localhost:58830/__testling?show=true:6944)
        at http://localhost:58830/__testling?show=true:9646
  ...

When trying to use testling, I copied example/* somewhere else and ran the testling command. It warned me about not having a "testling" field in the package.json, and directed me to "https://ci.testling.com/guide/quick_start", a broken link.

    ~/testlingexample testling
    The "testling" field isn't present in /home/fsantos/testlingexample/package.json.
    This field is required by testling. Please consult:
    https://ci.testling.com/guide/quick_start

Looked everywhere and tried everything to just run browserify test.js | testling, but I get the following error

/usr/local/lib/node_modules/testling/bin/cmd.js:215
ers && launch.browsers.local.filter(function(b){ return b.headless; })[0].name
                                                                         ^
TypeError: Cannot read property 'name' of undefined
    at ready (/usr/local/lib/node_modules/testling/bin/cmd.js:215:113)
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/testling/bin/cmd.js:92:26)
    at ChildProcess.EventEmitter.emit (events.js:98:17)
    at Process.ChildProcess._handle.onexit (child_process.js:789:12)

Any idea why?

One of my most favoritest things about testling is that it produces not only automated test results, but also you can view the tests locally in-browser. For libraries that interact with the DOM, it's great to also be able to visually verify your test cases.

Unfortunately, when you run testling with an html file (which makes it easier to test dom-based javascript), any externally loaded scripts are referenced from the root at which the testling command was run rather than where the file actually is. This means that if your tests are not at the root of your project, which they hardly ever are, you need a different set of paths for your tests to pass and for you to see the tests run locally. For example:

If you have this folder structure:

example-project
˪ package.json
˪ test
  ˪ index.js
  ˪ index.html

...and your html file has these contents...

<div class='foo'>wow such test</div>
<script src='index.js'></script>

...assuming that index.js contains your tests, if you were to open test/index.html, you would see your tests run and all would be well. However, if you were to run $ testling at the root of your project, assuming you have the package.json set up correctly, the tests would not run. If you then change the src of the script in your html file to test/index.js however, the tests will run.

Ideally, when testling ran using an html file, it would load any external paths based on the html file's base path rather than the base path that testling was run from 😀

I haven't looked into to testling's code much, but if you are busy and it would be easier for me to submit a PR, if you point me in the right direction I'd be happy to give this a shot!

The example test in ./example fails with an unhandled error.

Setup

node -v
v0.10.25
npm -v 
1.3.23
npm install -g testling # installs 1.5.9
npm install -g browserify # install 3.23.1
git clone [email protected]:substack/testling.git
cd testling && npm install

Test

browserify example/test.js | testling                                                                       

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: spawn ENOENT
    at errnoException (child_process.js:988:11)
    at Process.ChildProcess._handle.onexit (child_process.js:779:34)

Expected results are a the output in the README

console.log has no business being in a test, but ideally testling would warn rather than crash?

This caused a strange behavior where my tests passed on all versions of node (travis), but failed on all browsers (testling).

I'm having a difficult time getting my setup to work, although everything seems to be in order. I have PhantomJS installed:

-bash-4.1$ node node_modules/testling/node_modules/browser-launcher/example/detect.js
# available browsers:
[ { command: 'phantomjs',
    version: '1.9.7',
    name: 'phantom',
    re: /(\S+)/,
    type: 'phantom',
    headless: true,
    profile: true },
  { command: 'firefox',
    version: '24.3.0',
    name: 'firefox',
    re: /Mozilla Firefox (\S+)/,
    type: 'firefox',
    profile: true } ]

It works:

-bash-4.1$ phantomjs tests/index.js
Error: Cannot find module 'assert'

  phantomjs://bootstrap.js:289
  phantomjs://bootstrap.js:254 in require
  tests/index.js:13

but yet testling can't find it:

-bash-4.1$ make coverage
./node_modules/.bin/browserify -t coverify tests/index.js | testling | ./node_modules/.bin/coverify
No headless browser found.

Any suggestions?