axemclion / grunt-saucelabs Goto Github PK

See jmreidy/sauce-tunnel#12

Example: https://travis-ci.org/twbs/bootstrap/jobs/18113236

Tested http://127.0.0.1:3000/js/tests/index.html
Platform: undefined
>> Test Error
Url undefined

(I had to use process of elimination to determine that this error was from OS X Chrome.)

Thanks for this super-awesome plugin!

With auto-updating browsers like FireFox and Chrome releasing new major versions so frequently, it's hard to keep up with sauce configurations like this, which support the latest 2 releases:

{
  browserName: "firefox",
  version: "22",
  platform: "XP"
},
{
  browserName: "firefox",
  version: "21",
  platform: "XP"
}

Since SauceLabs knows what the latest version # of a browser is, It would be great if we could write our configurations like this:

{
  browserName: "firefox",
  version: "latest",
  platform: "XP"
},
{
  browserName: "firefox",
  version: "latest-1",
  platform: "XP"
}

Then we wouldn't have to rush to update our configuration every time a new major release happened.

When running saucelabs-mocha and SAUCE_ACCESS_KEY or SAUCE_USERNAME is not defined, the task fails without telling the user what is wrong:

$ grunt test

Running "connect:server" (connect) task
Started connect web server on 127.0.0.1:9999.

Running "saucelabs-mocha:all" (saucelabs-mocha) task
=> Connecting to Saucelabs ...
Warning: Task "saucelabs-mocha:all" failed. Use --force to continue.

Aborted due to warnings.

The task should print a helpful error message instead:

Running "saucelabs-mocha:all" (saucelabs-mocha) task
=> Connecting to Saucelabs ...
Authorization failed, make sure you have correctly
set SAUCE_USERNAME and SAUCE_ACCESS_KEY.

first i have to thank you for your grunt-plugin, which we are planning to use for our jasmine-tests. great effort!

we need the detailed error results (in case some tests fail). for jasmine reading the results is kind of easy (jasmine just outputs failing test-results).

i made a settable option (detailedError), made this available for the qunit and the jasmine-runner. even though the qunit log isnt that meaningful, it could still help.

i will create a pull request (issue #24)

Hey Parashuram,

What do you think about supporting a more compact browsers object, rather than having to explicitly list out every combination of browserName/platform/version?

For example: https://github.com/jstayton/jquery-marcopolo/blob/master/grunt.js#L103

I wrote that helper function because my project tests against almost all browsers, which ended up creating a huge array of different combinations. The compact style is much more manageable in my opinion.

Let me know what you think. Thanks!

It seems the public option is not working. I have a private sauce account. So by default my test are private.

When I pass

'public':'public',
build: new Date,

In the grunt config the build works, but the public option is ignored. Is this an issue with the plugin or saucelabs?

Not sure how hard this would be, but I think that a lot of people are gravitating toward Mocha because of its flexible API and its nice handling of async stuff, and I know I'd love to see it be easier to run Mocha tests on Saucelabs :)

https://saucelabs.com/docs/rest#jsunit
Most seemed to work with this repo's tests
YUI: https://saucelabs.com/tests/f55bbaf2f0f54b7e9229507267ac5c82
Jasmine: https://saucelabs.com/tests/d47342af981c42f59c766841a393b93b
QUnit: https://saucelabs.com/tests/4b9bcfd49a494a66a5b933e79c1993a8
Mocha: https://saucelabs.com/tests/c7162f57ed8249719ec26cd2237a3a7d seems to fail because it is expecting the mochaResults function from https://github.com/visionmedia/mocha-cloud

I had an issue where a number of my tests weren't running on certain browsers with the following error:

Could not read test result for Error: Error response status: 7

The error happened on these browsers:

• Mac 10.6::ipad(4.3)
• Windows 2012::internet explorer(10)
• Mac 10.6::ipad(4.3)
• Mac 10.6::ipad(5)
• Mac 10.6::iphone(4.3)
• Mac 10.6::ipad(5)
• Mac 10.6::iphone(5)
• Linux::android(4)

But always successfully ran the tests for the following browsers:

• Windows 2008::chrome
• firefox(18.0)

I was initially very confused because I had another project I was using grunt-saucelabs with that was working just fine, using the same browsers. The only difference was port 3000 and port 3003 - I tried switching from 3003 to 3000, all of my tests started running again! Didn't dig too deep into why this was happening, but filing it anyways.

Would be a big help to see a high-level summary of changes from version-to-version, especially when it comes to potentially breaking changes, or new APIs available.

Thanks!

Edit Changed from recommending README section to releases tab, per @nschonni 's suggestion below.

What version of jasmine does grunt-saucelabs work with?

I'm using the standard jasmine-html.js reporter with jasmine 1.2.0 and I'm getting the following exception:

Starting Jasmine tests
Fetched test result element, waiting for text inside it to change to complete

/Users/smithclay//node_modules/grunt-saucelabs/node_modules/wd/lib/webdriver.js:70
                var messageMatch = value.match(/([^\n]+)\nCommand duration/);
                                         ^
TypeError: Cannot call method 'match' of null
    at Object.<anonymous> (/Users/smithclay//node_modules/grunt-saucelabs/node_modules/wd/lib/webdriver.js:70:42)
    at Object.stringify (native)
    at /Users/smithclay//node_modules/grunt-saucelabs/node_modules/wd/lib/webdriver.js:65:31
    at Error.webdriver._newError.err.inspect (/Users/smithclay//node_modules/grunt-saucelabs/node_modules/wd/lib/webdriver.js:92:9)
    at formatValue (util.js:197:25)
    at inspect (util.js:125:10)
    at Object.exports.format (util.js:50:20)
    at Object.exports.log (console.js:25:36)
    at module.exports.TestRunner.jasmineRunner (/Users/smithclay//node_modules/grunt-saucelabs/tasks/saucelabs.js:204:16)
    at webdriver._callbackWithData (/Users/smithclay/Dropbox/Work/server/node_modules/grunt-saucelabs/node_modules/wd/lib/webdriver.js:193:29)

It seems like this is because webdriver is trying to get the element text of an element on the page that doesn't exist. Here's the saucelabs raw log output:

17:42:47.427 INFO - Executing: [get text:  null] at URL: /session/1354326154480/element//text)
17:42:47.427 WARN - Exception thrown
java.lang.NullPointerException
    at org.openqa.selenium.remote.server.handler.GetElementText.call(GetElementText.java:30)
    at org.openqa.selenium.remote.server.handler.GetElementText.call(GetElementText.java:1)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
17:42:47.434 WARN - Exception: null

Every so often, this plugin fails on Travis, with no particular error:

Running "saucelabs-mocha:all" (saucelabs-mocha) task
Verifying property saucelabs-mocha.all exists in config...OK
File: [no files]
=> Connecting to Saucelabs ...
Warning: Task "saucelabs-mocha:all" failed.� Use --force to continue.
Aborted due to warnings.
npm ERR! weird error 3
npm ERR! not ok code 0
The command "npm test" exited with 1.
Done. Your build exited with 1.

Here is the build URL:
https://travis-ci.org/ModelN/backbone.geppetto/builds/12358997

Perhaps it was a temporary connectivity issue with SauceLabs? I'd like to run this task as a prerequisite for every pull request on my projects, if possible.

Thanks for any help or advice!

In my project I got some pending tests (tests that I didn't have time to write yet):

They make the tasks nastily fail:

...
Running "saucelabs-mocha:all" (saucelabs-mocha) task
=> Connecting to Saucelabs ...
>> 2013-09-24 20:25:20.006:INFO::jetty-7.x.y-SNAPSHOT
>> 2013-09-24 20:25:20.035:INFO::Started [email protected]:55564
>> 2013-09-24 20:25:52.313:INFO::jetty-7.x.y-SNAPSHOT
>> 2013-09-24 20:25:52.315:INFO::Started [email protected]:4445
>> Connected to Saucelabs
Fatal error: Cannot read property '1' of null

Seems like your latest release isn't up on NPM!

Error: version not found: 2.0.3 : grunt-saucelabs/2.0.3

It would be great to have control over the version of selenium used on sauce labs
Sauce labs suppors it with the selenium-version directive.

Wish I could give a more detailed response of what exactly is happening. As of now, all I know is that mocha tests are reporting incorrectly.

Example (Screencast shows 28 passing tests):
https://saucelabs.com/tests/7dd60827dd0a469bb9473ea6422cb8b9

Possibly related to the failing tests here:
https://saucelabs.com/u/grunt-sauce

Thanks!

Hey Parashuram,

This has been working great for me — thanks for your work on it!

Question: Is there a way to send the build ID and set the test pass/fail status? Looking at the Sauce Labs Test Dashboard, that's information that would be really handy to see.

Thanks!

Hello!

This plugin is awesome, but I'm running into an intermittent issue when trying to run my mocha tests on SauceLabs.

When I run locally, sometimes the tests pass and other times they fail in random places:

>> Connected to Saucelabs


Tested http://localhost:9001/specs/index.html
Environment: Backbone.Geppetto
>> Result: passes: 43failures: 0duration: 0.59s
Test Video: http://saucelabs.com/tests/ef93cc00e98042eea9f2b99d8ce872f4


Tested http://localhost:9001/specs/index.html
Environment: Backbone.Geppetto
>> Result: passes: 43failures: 0duration: 0.48s
Test Video: http://saucelabs.com/tests/4962d37ed3e8481faddd39494a3de619


Tested http://localhost:9001/specs/index.html
Environment: Backbone.Geppetto
>> Result: passes: 43failures: 0duration: 0.23s
Test Video: http://saucelabs.com/tests/842ab16a911d44aeb5fbdd51f5537bf1


Tested http://localhost:9001/specs/index.html
Environment: Backbone.Geppetto
>> Result: passes: 43failures: 0duration: 0.54s
Test Video: http://saucelabs.com/tests/2d36313c003045c2adf379f42d2d9585


Tested http://localhost:9001/specs/index.html
Environment: Backbone.Geppetto
>> Result: passes: 43failures: 0duration: 0.24s
Test Video: http://saucelabs.com/tests/f4219406ccfc492890a337def7174ce8


Tested http://localhost:9001/specs/index.html
Environment: Backbone.Geppetto
>> Result: passes: 0failures: 0duration: 0s
Test Video: http://saucelabs.com/tests/9c21a53bdccc4003b1065e496bf0b550
>> [Backbone.Geppetto] Could not read test result for Error: Error response status: 7.  http://localhost:9001/specs/index.html
>> [Backbone.Geppetto] More details at http://saucelabs.com/tests/http://localhost:9001/specs/index.html
Fatal error: Cannot read property '2' of undefined

I also tried running the tests on Travis-CI and got a similar problem. See: https://travis-ci.org/ModelN/backbone.geppetto/builds/11800015

Any ideas would be much appreciated!

I've been trying to get my CI working again and after fixing up against some changes to the task usage I my Jasmine tests don't seem to finish.

Here's my build job and each time my Source Labs connection times out. I've looked at Source Labs and the jobs aren't getting marked as passed or fail set on them, which means that the task just times out.

I've setup with the jsrunner but still that doesn't seem to have helped, I'm not sure if I've missed something else on the task setup.

jasmine-jsreporter doesn't seem to support jasmine 2.0.0 yet, but @sclevine submitted a PR which seems to have done most of the heavy lifting - see detro/jasmine-jsreporter#7

I can get sauce running the tests by including the modified jasmine-jsreporter inside our Gruntfile's grunt-contrib-jasmine vendor section, which overwrites the JSReporter from grunt-saucelabs, and namespaces window.jasmine.getJSReport().

The screencast then says "625 specs, 0 failures, 9 pending specs" and is green/passing, and the selenium log says all suites and specs are passing, however sauce (or grunt-saucelabs?) is saying the tests haven't passed. Going to the JSUnitTest tab on the sauce job however tells me Jasmine run on Sauce failed: 0 total: 0 passed: 0, finished in s, and Passing 0 tests.

Is Jasmine 2 support on the grunt-saucelabs roadmap? If so what are the blockers currently?

Currently, the tests are checked for completion every 5000 milli seconds. This could be also be a configuration value passed to the test suite.

grunt-saucelabs has evolved a lot since last npm publish: bug fix and so. But if nobody does a version-bump and a publish, grunt-saucelabs package will be buggy for ever.

I noticed that if you do not set the options.testname optional parameter, the output will include the proper operating system and browser (http://cl.ly/image/0e3I3D3Q3j19) while rendering the web-based output's 'session' column with "unnamed job" (http://cl.ly/image/1A2M1d0f3n3Q). Is there a way to add that as a variable to be interpolated by saucelabs (Example: "My Project: #{TEST_ENVIRONMENT}") while maintaining both the console output and web-based session column?

So, testInterval's description alludes to there being retry logic, but no retrying actually seems to take place:

Running "saucelabs-qunit:all" (saucelabs-qunit) task
=> Connecting to Saucelabs ...
>> 2014-01-17 01:34:07.169:INFO::jetty-7.x.y-SNAPSHOT
>> 2014-01-17 01:34:07.197:INFO::Started [email protected]:49843
>> 
>> 2014-01-17 01:35:05.935:INFO::jetty-7.x.y-SNAPSHOT
>> 2014-01-17 01:35:05.939:INFO::Started [email protected]:4445
>> Connected to Saucelabs

[...other successful tests...]

Tested http://127.0.0.1:3000/js/tests/index.html
Environment: OS X 10.9::chrome
>> Timeout, waited for more than 300000 milliseconds
>> All tests completed with status false
=> Killing tunnel d04fee418fb94166a98e4afceacbfbf4
Warning: Task "saucelabs-qunit:all" failed.� Use --force to continue.

(OS X Chrome 31 seems to be flaky lately for some reason.)

[Edit: Okay, so what I'm really looking for is a "have each test (page+browser combo) timeout after X amount of time and retry each timed-out/failed test up to N times" feature.]

I see in the Gruntfile of this repository that it uses "googlechrome".

On https://saucelabs.com/docs/browsers I see "chrome" in the webdriver/nodejs documentation, and "googlechrome" in the selenium/json documentation.

I see that this grunt plugin is using webdriver, so I figured it should be "chrome", right?

Or does both work and is the documentation at saucelabs simply confusing?

Rather than repeating the host/port combination in the urls object, what about supporting the default Grunt multiTask files object.

'saucelabs-qunit': {
  all: {
      options: {
        host: 'http://localhost',
        port: 8000,
        // rest of the regular config
      }
      src: [ 'test1.html', 'test2.html'}
    }
  }
}

This would then just iterate over the files object and build up the current options.urls array

Saucelabs tests cannot be run since the saucelabs key and value are not available. Just need to run jshint for pull requests. All pull requests need to be sent to incoming-pr branch that will then run the saucelabs tests.

since grunt-saucelabs has, probably, the best WD runners for js test frameworks, maybe we should implement the optional local test runner on local browsers (if SAUCE_USERNAME is empty and username is empty, for example)

Running my tests auto-magically on Sauce Labs through Grunt would be awesome. Unfortunately, I seem to be running into an issue after the connection is established:

2013-01-18 15:21:20,940 - Connected! You may start your tests.
Fatal error: Cannot read property '_' of undefined

This also happens even if I enter wrong configuration information (empty API key):

=> Sauce Labs Tunnel disconnected  3
Fatal error: Cannot read property '_' of undefined

I'm guessing this may be an issue with my version of Grunt - I'm on grunt v0.4.0rc6, and grunt-contrib-qunit 0.1.1rc6.

Would love to get some feedback on this!

It would be great to add an option that passes the additional feature configuration such as:

"video-upload-on-pass": false

I keep getting a timed out message from saucelabs for modern browsers:

{"browserName": "internet explorer","platform": "Windows 7","version": "9"},
{"browserName": "firefox","platform": "OS X 10.6"},
{"browserName": "chrome","platform": "Windows XP"}

The test runs: https://saucelabs.com/tests/82f5c00caa10443ca80e3b20e68570b4

I committed a ticket to them asking if this was an issue on there side. They responded and said that it was not.

The output from the console is:

Environment: Unit Tests

Timeout, waited for more than 300000 milliseconds
All tests completed with status false
Warning: Task "saucelabs-mocha:all" failed. Use --force to continue.

Can you guys help me understand if this is an issue with the plugin or my configuration?

thanks, cody

QUnit provides callbacks like QUnit.done that can be used instead of querying the DOM for result elements, assuming the driver allows execution of arbitrary code along with a callback mechanism.

I'm using grunt-saucelabs v4.1.2 in a grunt project.

Gruntfile.js:

var browsers = [
  ...
    { browserName: "android", platform: "Linux", version: "4.0" }
];

grunt.initConfig({
  ...
  saucelabs-qunit': {
        all: {
            options: {
                username: 'iris-js',
                key: process.env.SAUCE_ACCESS_KEY,
                urls: ["http://localhost:8080/test/iris.html"],
                tunnelTimeout: 5,
                build: process.env.TRAVIS_JOB_ID,
                concurrency: 3,
                browsers: browsers,
                testname: "qunit tests from saucelabs"
            }
        }
    }
});

When I run SAUCE_ACCESS_KEY=*** grunt test-saucelabs, It only fails on Android tests... Saucelabs screencast shows a blank screen: 'HTTP ERROR 404'. In other platforms (Linux, Windows, iOS, OS X...) all goes well...

There are exceptions in the selenium log:

...
java.lang.UnsupportedOperationException: Window handling not supported on Android
...
org.openqa.selenium.NoSuchElementException: Could not find element with id: qunit-testresult

Why does tests only fail with Android? I'm trying to find a solution but I cannot find it... What am I doing wrong?

Thanks in advance

I would actually love to be able to use this with mocha, re #4.

Hi, I had a personal saucelabs account and I set the username & key in the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables and everything works when I run the grunt saucelabs task.

Then created an open source account and set the new username and key in my Gruntfile.js without changing my environment variables. After this, all of my tests started failing like so:

>> [SwfStore] Could not initialize browser for session undefined { browserName: 'internet explorer',

And the tests never even appeared in my account page on the saucelabs website.

However, everything passed when I re-ran the tests like so:

 $ SAUCE_USERNAME=new_username SAUCE_ACCESS_KEY=new_key grunt test

So I think some part of grunt-saucelabs code is using the ENV variables, even when the grunt options have a username and key set.

Here's the full output:

$ grunt test
Running "saucelabs-jasmine:all" (saucelabs-jasmine) task
=> Connecting to Saucelabs ...
>> 2013-12-17 12:47:45.627:INFO::jetty-7.x.y-SNAPSHOT
>> 2013-12-17 12:47:45.683:INFO::Started [email protected]:52243
>> 2013-12-17 12:48:35.790:INFO::jetty-7.x.y-SNAPSHOT
>> 
>> 2013-12-17 12:48:35.794:INFO::Started [email protected]:4445
>> Connected to Saucelabs
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'internet explorer',
>>   version: '11',
>>   platform: 'Windows 8.1',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[33mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'internet explorer',
>>   version: '10',
>>   platform: 'Windows 8',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[36mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'internet explorer',
>>   version: '9',
>>   platform: 'Windows 7',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[35mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'internet explorer',
>>   version: '8',
>>   platform: 'Windows XP',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[34mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'internet explorer',
>>   version: '7',
>>   platform: 'Windows XP',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[32mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'internet explorer',
>>   version: '6',
>>   platform: 'Windows XP',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[31mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'firefox',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[33mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'chrome',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[36mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'safari',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[35mSwfStore\u001b[39m' }
>> [SwfStore] Could not initialize browser for session undefined { browserName: 'opera',
>>   tags: [],
>>   name: 'SwfStore',
>>   build: undefined,
>>   'tunnel-identifier': '156534453',
>>   prefix: '\u001b[34mSwfStore\u001b[39m' }
>> All tests completed with status false
=> Killing tunnel 46f347c9a97a4d3f88b21c13b57bd0d0
Warning: Task "saucelabs-jasmine:all" failed. Use --force to continue.

Aborted due to warnings.

Here's the sauce_connect.log from a failing run, in case it helps:

2013-12-17 12:47:45,714 - sauce_connect:558 - INFO - / Starting \
2013-12-17 12:47:45,721 - sauce_connect:559 - INFO - Please wait for "You may start your tests" to start your tests.
2013-12-17 12:47:45,733 - sauce_connect:571 - DEBUG - System is -4.0 hours off UTC
2013-12-17 12:47:45,734 - sauce_connect:573 - DEBUG - options: {'user': 'jsfc', 'ports': ['52243'], 'domains': None, 'debug_ssh': False, 'tunnel_ports': ['80'], 'allow_unclean_exit': False, 'rest_url': 'https://saucelabs.com/rest/v1', 'logfile': 'sauce_connect.log', 'squid_opts': '', 'ssh': False, 'se_port': '4445', 'readyfile': None, 'shared_tunnel': False, 'boost_mode': True, 'tunnel_identifier': '156534453', 'ssh_port': 443, 'fast_fail_regexps': '', 'direct_domains': '', 'host': '127.0.0.1', 'quiet': False, 'latency_log': 150, 'use_ssh_config': False}
2013-12-17 12:47:45,736 - sauce_connect:574 - DEBUG - metadata: {'PythonVersion': '2.5.1', 'OwnerHost': '127.0.0.1', 'Release': '3.0-r28', 'OwnerPorts': ['52243'], 'Ports': ['80'], 'Platform': 'Java-1.6.0_65-Java_HotSpot-TM-_64-Bit_Server_VM,_20.65-b04-462,_Apple_Inc.-on-Mac_OS_X-10.9-x86_64', 'Build': '43', 'ScriptRelease': 43, 'ScriptName': 'sauce_connect'}
2013-12-17 12:47:45,737 - sauce_connect:576 - INFO - Forwarding: None:['80'] -> 127.0.0.1:['52243']
2013-12-17 12:47:45,752 - sauce_connect:375 - INFO - Succesfully connected to local server 127.0.0.1:52243 in 5ms
2013-12-17 12:47:46,698 - sauce_connect:238 - INFO - {"squid_config":null,"use_caching_proxy":true,"metadata":{"PythonVersion":"2.5.1","OwnerHost":"127.0.0.1","Release":"3.0-r28","OwnerPorts":["52243"],"Ports":["80"],"Platform":"Java-1.6.0_65-Java_HotSpot-TM-_64-Bit_Server_VM,_20.65-b04-462,_Apple_Inc.-on-Mac_OS_X-10.9-x86_64","Build":"43","ScriptRelease":43,"ScriptName":"sauce_connect"},"use_kgp":true,"tunnel_identifier":"156534453","shared_tunnel":false,"fast_fail_regexps":null,"ssh_port":443,"direct_domains":null,"domain_names":[]}
2013-12-17 12:47:54,276 - sauce_connect:250 - INFO - Tunnel remote VM is provisioned (46f347c9a97a4d3f88b21c13b57bd0d0)
2013-12-17 12:47:54,891 - sauce_connect:268 - INFO - Tunnel remote VM is booting ..
2013-12-17 12:48:35,663 - sauce_connect:272 - INFO - Tunnel remote VM is running at maki76236.miso.saucelabs.com
2013-12-17 12:48:35,677 - sauce_connect:375 - INFO - Succesfully connected to local server 127.0.0.1:52243 in 1ms
2013-12-17 12:48:35,681 - sauce_connect:646 - INFO - Starting connection to tunnel host...
2013-12-17 12:48:35,684 - sauce_connect:646 - INFO - Connecting to tunnel host maki76236.miso.saucelabs.com as jsfc
2013-12-17 12:48:35,786 - sauce_connect:646 - INFO - Forwarding Selenium with ephemeral port 52256
2013-12-17 12:48:35,796 - sauce_connect:646 - INFO - Selenium HTTP proxy listening on port 4445
2013-12-17 12:48:36,486 - sauce_connect:0 - INFO - Successful handshake with Sauce Connect server
2013-12-17 12:48:36,601 - sauce_connect:0 - INFO - Tunnel host version: 0.1.0, remote endpoint ID: 477c14f9c94943a5b96bb0cc60bc9fb9
2013-12-17 12:48:36,605 - sauce_connect:646 - INFO - Connected! You may start your tests.
2013-12-17 12:48:50,417 - sauce_connect:0 - INFO - received SIGTERM

My tests run at saucelabs (via saucelabs-mocha), and all pass in the test runner (https://www.dropbox.com/s/3n5v6eqxlkwt1r8/Screenshot%202013-10-02%2012.51.44.png), but this is what I get in the command line.

Note133:UX360 tandemseven$ grunt saucelabs
Running "saucelabs-mocha:all" (saucelabs-mocha) task
=> Connecting to Saucelabs ...

Connected to Saucelabs
Error - Could not extract passes, failures, or duration from text TypeError: Cannot read property '1' of null
Fatal error: Cannot read property '2' of undefined

What am I doing wrong here?

Right now when a test fails (with detailedErrors: true) ,grunt-saucelabs just dumps the entire qunit test results dom as a single string, coloured red, into the console. Which means a potentially very long list of test results is just outputted in bright red (test results of tests that succeeded as well as tests that failed).

It would be much more useful if it only outputted the results of failed tests, or at least distinguished between the two by color (or something).

So that travis can use encrypted env variables

Using the standard this.options() inside of your Grunt multi tasks will allow task-level options objects to automatically be merged and overridden by target-level options objects. It can make the configuration of the task much more idiomatic.

http://gruntjs.com/api/inside-tasks#this.options

I'm posting this issue to let you know that we will be publishing Grunt 0.4 on Monday, February 18th.

If your plugin is not already Grunt 0.4 compatible, would you please consider updating it? For an overview of what's changed, please see our migration guide.

If you'd like to develop against the final version of Grunt before Monday, please specify "grunt": "0.4.0rc8" as a devDependency in your project. After Monday's release, you'll be able to use "grunt": "~0.4.0" to actually publish your plugin. If you depend on any plugins from the grunt-contrib series, please see our list of release candidates for compatible versions. All of these will be updated to final status when Grunt 0.4 is published.

Also, in an effort to reduce duplication of effort and fragmentation in the developer community, could you review the grunt-contrib series of plugins to see if any of your functionality overlaps significantly with them? Grunt-contrib is community maintained with 40+ contributors—we'd love to discuss any additions you'd like to make.

Finally, we're working on a new task format that doesn't depend on Grunt: it's called node-task. Once this is complete, there will be one more conversion, and then we'll never ask you to upgrade your plugins to support our changes again. Until that happens, thanks for bearing with us!

If you have any questions about how to proceed, please respond here, or join us in #grunt on irc.freenode.net.

Thanks, we really appreciate your work!

Would be good to add the saucelabs URL to the log files, after the test has finished. IF there are failures, it would help look at the right test run.

Additionally, if there a way to make all test run URLs public ?

See https://travis-ci.org/twbs/bootstrap/jobs/17089345

Running "saucelabs-qunit:all" (saucelabs-qunit) task
=> Connecting to Saucelabs ...
=> Looks like there are existing tunnels to Sauce Labs, need to kill them. TunnelID:<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/0.7.62</center>
</body>
</html>
=> Killing tunnel <
=> Killing tunnel !
=> Killing tunnel D
=> Killing tunnel O
=> Killing tunnel C
=> Killing tunnel T

The browser object can be exposed to the callback so that the callback can get reports, or run additional diagnostics once a test is complete.

I am trying the mocha example from your website https://saucelabs.com/docs/javascript-unit-testing-tutorial

And I get the following result:

gmoon:yummy-pasta gmoon$ grunt test
Running "connect:server" (connect) task
Starting connect web server on localhost:9999.

Running "saucelabs-mocha:all" (saucelabs-mocha) task
=> Connecting to Saucelabs ...
=> Looks like there are existing tunnels to Sauce Labs, need to kill them. TunnelID:a6efe879355247e8a9fb2db3fb8316f0
=> Killing tunnel a6efe879355247e8a9fb2db3fb8316f0
>> 2013-08-24 16:13:24.132:INFO::jetty-7.x.y-SNAPSHOT
>> 2013-08-24 16:13:24.165:INFO::Started [email protected]:57941
>>
>> 2013-08-24 16:14:09.664:INFO::jetty-7.x.y-SNAPSHOT
>>
>> 2013-08-24 16:14:09.668:INFO::Started [email protected]:4445
>> Connected to Saucelabs
Fatal error: Cannot read property '1' of null

This is due to an apparent change in the mocha.suite.total() result. The code is expecting to match this:

/passes: (\d*)failures: (\d*)duration: ([\d,.]*)s/

But the return value now contains a 'pending' section:

passes: 5pending: 0failures: 0duration: 0.02s

I created a pull request with a fix.