philipwalton / easy-sauce Goto Github PK

I've noticed that when running tests using Firefox on both Mac (10.11) and Windows (10), the test results never return. Firefox on Linux doesn't have this issue.

I can open the dashboard and watch live the tests executing and passing for Mac and Windows, but the session continues on for almost 7 minutes with no results reported. Eventually, the session disconnects and reports that there were test failures.

Browser Config:

[ [ "Windows 10", "firefox", "latest" ],
  [ "Mac 10.11", "firefox", "latest" ],
  [ "Linux", "firefox", "latest" ]
]

Console report:

Saucelabs Dashboard (Linux session success, the other two still waiting):

Live screenshot from the open test window:

Not sure what other information may be helpful here, but glad to provide more if needed. Have you experienced the same?

Is there a way to configure how many tests can run at a time with this plugin? How would you go about doing that?

In my package.json file I have the test script set to easy-sauce:

{
  "scripts": {
    "test": "easy-sauce"
  }
}

And in node environment 5.6, I get the following error when trying to run npm test:

/data/projects/personal/easytest/node_modules/easy-sauce/lib/cli.js:114
function formatResult(result = {}) {
                             ^

SyntaxError: Unexpected token =
    at exports.runInThisContext (vm.js:53:16)
    at Module._compile (module.js:387:25)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (/data/projects/personal/easytest/node_modules/easy-sauce/bin/easy-sauce:6:13)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)

Works fine in node 6, but in the project I'd like to use this in, we can't use node 6 yet. What's the best way to resolve this?

So I have this:

const path = require('path');
const easySauce = require('easy-sauce');

easySauce({
  username: process.env.SAUCE_LABS_USERNAME,
  key: process.env.SAUCE_LABS_KEY,
  testPath: path.resolve(process.cwd() + '/test/unit/one.test.html'),
  platforms: [
    [
      'Windows 10',
      'chrome',
      'latest'
    ],
    [
      'Linux',
      'firefox',
      'latest'
    ],
    [
      'OS X 10.11',
      'safari',
      '9'
    ]
  ]
})
.on('message', function(message) {
  // A message has been emitted, inform the user.
  console.log(' => Message => ',message);
})
.on('update', function(job) {
  // A job's status has been updated
  console.log('=> Update => ',job.status);
})
.on('done', function(passed, jobs) {
  // All tests have completed!
  if (passed) {
    console.log('All tests passed!');
  }
  else {
    console.log('Oops, there were failures:\n' + jobs);
  }
})
.on('error', function(err) {
  // An error occurred at some point running the tests.
  console.error(err.message);
});

And I get this output

 => Message =>  Started static web server on port 8080
 => Message =>  Created secure tunnel to 8080
 => Message =>  Running tests for https://jowyggxzmz.localtunnel.me/Users/alexamil/WebstormProjects/cisco/cdt-now/test/unit/one.test.html on Sauce Labs.
=> Update =>  test queued
=> Update =>  test queued
=> Update =>  test queued
=> Update =>  test session in progress
=> Update =>  test session in progress
connect ETIMEDOUT 162.222.75.243:443
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: connection refused: localtunnel.me:44967 (check your firewall settings)
    at Socket.<anonymous> (/Users/alexamil/WebstormProjects/cisco/cdt-now/node_modules/localtunnel/lib/TunnelCluster.js:47:32)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1281:8)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

Do you happen to know if this type of firewall issue is common and if there is a usual workaround?

thanks!

Hey

With the Node.js API, it's not clear whether it uses the easySauce config in the package.json file.
Specifically, I don't know how to specify the test.html file to use, if we do not have the config in package.json. I have no problem using the easySauce property in package.json, but I think maybe the Node.js API example should be self-contained enough to include the path to the html file?

Problem

If you pass tests: "test/index.html", the URL generated is:

https://470827b6.ngrok.iotest/index.html

Solution

Leverage url module to format the URL accordingly from the ngrok hostname and provided path.

Problem is related to ngrok. It allows only 20 connections per 1 minute. If I have e.g. 10 platforms then tests on some of them will be failed because of to many connections which are rejected by ngrok.
Thus I propose to make POLL_INTERVAL one of the settings param. So I know that I have a lot of platforms and poll POLL_INTERVAL should be increased.
Another idea - make it smart. If platforms number is big the Increase it automatically. But I'm not sure about all mechanism and stuff...

Currently, it's defaulted to ./ for ngrok.

The problem is, our test suite is built with Webpack (as the client-side app is) and put into /build alongisde /build/test/index.html.

If I use tests: "/build/test/index.html, the publicPath for Webpack is / (/build is stripped out).

This means that instead of 👍

$ easy-sauce -c easysauce.json

I have to change directories:

$ cd build && easy-sauce -c ../easysauce.json

It seems a simple solution, but when spinning up machines & waiting for tests to pass/fail to see that the problem is primarily with path issues was a bit painful.

I'm more than happy to submit a PR if you'd like.

Thanks!

Would you be interested in adding a more flexible syntax to specify the platforms/browsers/versions? For example:

{
  "platforms": [
    {
      "name": "internet explorer",
      "version": "oldest..latest"
    },
    {
      "name": "chrome",
      "version": ["oldest", "latest"]
    }
  ]
}

What I'm thinking is the same syntax used in zuul. The advantage is that you don't have to update the config if a new version of a browser is added in SauceLabs. The same applies for removed versions.

Converting this syntax to the required format is easy (the code can be extracted from zuul) and can be done in a separate npm module.

The only problem I see is that it isn't backward compatible.

Thoughts?