Requires DevExpress/testcafe-hammerhead#25 resolution

In addtion to Travis it would be nice to have AppVeyor tests for server, so will be able to test Windows specific things. @AlexanderMoskovkin Is it possible to teach our bot accumulate test results from both Travis and AppVeyor?

We need some kind of spike project which will generate message for all errors in context of different reporters to figure out best coloring for the ttyDecorator.
\cc @kirovboris @VasilyStrelyaev

We can check that the module works with this

1. runner.src([]).run() - should not open a browser until all tests are not compiled. Because if compilation fails we end up with the hanging open local browser connections.
2. runner.src(someTests).filter(()=>false).run() - runner crashes with a js error now

As a continuation of the DevExpress/testcafe-hammerhead#65 we need to remove jobUid and jobOwnerToken from the client code and replace it with the sessionId. We need to do that for the tests and server (if we have anything there) as well.

From Migrating to 1.0.0:
The most important difference in v1.0.0 is that all rules are off by default. When using --init, your configuration file will automatically include the following line:

{
    "extends": "eslint:recommended"
}

This setting mimics some of the default behavior from 0.x, but not all.
These rules are no longer on by default, so you should review your settings to ensure they are still as you expect:
no-alert
no-array-constructor
no-caller
no-catch-shadow
no-empty-label
no-eval
no-extend-native
no-extra-bind
no-extra-strict
no-implied-eval
no-iterator
no-label-var
no-labels
no-lone-blocks
no-loop-func
no-multi-spaces
no-multi-str
no-native-reassign
no-new
no-new-func
no-new-object
no-new-wrappers
no-octal-escape
no-process-exit
no-proto
no-return-assign
no-script-url
no-sequences
no-shadow
no-shadow-restricted-names
no-spaced-func
no-trailing-spaces
no-undef-init
no-underscore-dangle
no-unused-expressions
no-use-before-define
no-with
no-wrap-func
camelcase
comma-spacing
consistent-return
curly
dot-notation
eol-last
eqeqeq
key-spacing
new-cap
new-parens
quotes
semi
semi-spacing
space-infix-ops
space-return-throw-case
space-unary-ops
strict
yoda
See also: the full diff where the defaults were changed.

List of recommended rules:
comma-dangle - disallow or enforce trailing commas (recommended)
no-cond-assign - disallow assignment in conditional expressions (recommended)
no-console - disallow use of console in the node environment (recommended)
no-constant-condition - disallow use of constant expressions in conditions (recommended)
no-control-regex - disallow control characters in regular expressions (recommended)
no-debugger - disallow use of debugger (recommended)
no-dupe-args - disallow duplicate arguments in functions (recommended)
no-dupe-keys - disallow duplicate keys when creating object literals (recommended)
no-duplicate-case - disallow a duplicate case label. (recommended)
no-empty-character-class - disallow the use of empty character classes in regular expressions (recommended)
no-empty - disallow empty statements (recommended)
no-ex-assign - disallow assigning to the exception in a catch block (recommended)
no-extra-boolean-cast - disallow double-negation boolean casts in a boolean context (recommended)
no-extra-semi - disallow unnecessary semicolons (recommended)
no-func-assign - disallow overwriting functions written as function declarations (recommended)
no-inner-declarations - disallow function or variable declarations in nested blocks (recommended)
no-invalid-regexp - disallow invalid regular expression strings in the RegExp constructor (recommended)
no-irregular-whitespace - disallow irregular whitespace outside of strings and comments (recommended)
no-negated-in-lhs - disallow negation of the left operand of an in expression (recommended)
no-obj-calls - disallow the use of object properties of the global object (Math and JSON) as functions (recommended)
no-regex-spaces - disallow multiple spaces in a regular expression literal (recommended)
no-sparse-arrays - disallow sparse arrays (recommended)
no-unreachable - disallow unreachable statements after a return, throw, continue, or break statement (recommended)
use-isnan - disallow comparisons with the value NaN (recommended)
valid-typeof - Ensure that the results of typeof are compared against a valid string (recommended)
no-fallthrough - disallow fallthrough of case statements (recommended)
no-octal - disallow use of octal literals (recommended)
no-redeclare - disallow declaring the same variable more than once (recommended)
no-delete-var - disallow deletion of variables (recommended)
no-undef - disallow use of undeclared variables unless mentioned in a /*global */ block (recommended)
no-unused-vars - disallow declaration of variables that are not used in the code (recommended)
no-mixed-spaces-and-tabs - disallow mixed spaces and tabs for indentation (recommended)

We need to review .eslintrc for inheriting default rules and manually adding rules that were excluded from defaults (e.g. no-undef, semi and so).

await * didn't make it: https://github.com/tc39/tc39-notes/blob/master/es7/2015-07/july-30.md#64-advance-async-functions-to-stage-2 😞 So, most likely it will be removed in the future versions of Babel.

Instead of

await * arr;

use

await Promise.all(arr)

\cc @churkin @AlexanderMoskovkin Please, check the presence of this syntax in the hammerhead and browser-natives.

@kirovboris @AndreyBelym @LavrovArtem @miherlosev @arubtsov @helen-dikareva - just FYI

AFAIK we need this only for the Studio, so let pass this functionality to the decorators (and don't do it in the default decorators

)

Now we do it in the gulp task

• We should pass user-provided errorDecorator to the format, currently it's called formatter in the runner and reporter
• If decorator is not provided fallback to the tty or plain-text decorators (depends on the type of the output stream)
• Put decorators into the decorators dir in the sources
• Consider moving src/test-error content to the src/reporters/errors
• Rename src/test-error/index.js to the src/reporters/errors/format.js
• Test error diffs and viewportWidth since we have a wordwrap for text (https://github.com/superroma/testcafe-phoenix/blob/master/src/reporters/base.js#L140). Seems like we will need more complicated logic for the errors to consider indents.

Build bot should recognize the issue with a special 'release' title and create a comment in the issue with list of issues closed after the last release and the guide 'what you should do before release'

see: https://github.com/superroma/testcafe-hammerhead/blob/master/Gulpfile.js#L135

When several test files from the runner.src(files) argument require the same file, Compiler reads this file for each test file (but should read it once, save it to the cache and then read from the cache).
It occurs because of Compiler process all files in parallel.

The task should build sources before publishing and ensure that the NPM user is logged as devexpress. Then it should execute npm publish. It can perform additional checks, e.g. what CHANGELOG is updated, latest travis build is passing, etc.

• Get rid of util jquery functions using
• Get rid of jquery using for create, select and manipulate with dom-elements
• Get rid of jquery binding for 'ready' and 'load' events
• Get rid of jquery

There are couple of things I don't like in our client right now and which we should rethink before new API implementation:

• Create concept of autowait which will include current action-barrier code, page loads etc. and make it optional
• I don't understand current responsibilities of the client/runner/runner.js, currently it looks like a god-object and existence of the test-iterator makes whole thing more complicated. We need new and more clear runner structure
• Action descriptor concept in automations is too complicated and I already forgot why we need this. We need clean API for the automations, we need to refactor them as well.

1. Linting (scripts and tests)
2. Constants (const keyword, new naming format)
3. Classes (use where possible, any object which has state now or init method or something like this should be turned into the classes and instantiated there necessary)
4. Arrows
5. Promises (get rid of the async module and callbacks)

There are two ways which leads to the job completion:

1. All job's test runs are finished.
2. Job's task was aborted

In the first scenario browser requests status an we query job in the front of the queue for the next test run. If job doesn't contain pending test runs any more it returns null and raises done event. done event handler in the BrowserConnection removes job from the queue. But since BrowserConnection.getStatus() already queried queue browser goes to the idle state. It will query status anyway, however it leads to the unecessary delay.

The second scenario has more complicated problems. Task calls abort() for each of it's jobs. abort() just cleans pending test runs queue. So, from the browsers perspective we will have the same issue as described for the first case. The other issue for this approach is that we still relay on done event for the abortion scenario which is really fragile. done event on all jobs will trigger done event for the task. Which is wrong. Currently we rely on the fact that adopting code will call task.removeAllListeners() immediately after task.abort() which is definetly uncool.

The proposal is:

• Do not rely on the done event of the job in the BrowserConnection. Use the following algorithm instead:
  1. If jobQueue is empty then switch browser to the idle state and abort these steps.
  2. Try query job in the front of the queue. If it returns null then shift the queue and repeat this step. Else pass returned test run to the browser.
• task.abort() should immediately remove all task's jobs from the appropriate browser connections.

Split scripts to:

• core.js (or testcafe_core.js)
• runner.js (or testcafe_runner.js)
• ui.js (or testcafe_ui.js)

Each module has documented API. Modules are available via a global variable in window.

/cc @inikulin

Is it really usable to keep test running if assertion is already failed?

Pros:

• Simplify our code (no more fatal/non-fatal errors)
• Cleaner reports
• Possibly faster test runs

Let's discus \cc @churkin @AlexanderMoskovkin @kirovboris @miherlosev @LavrovArtem @arubtsov @AndreyBelym @VasilyStrelyaev @MargaritaLoseva

Something like expect(assertion_count)

It's necessary to use the same module with the Promise class on the client and server sides. The promise module have no implementation for the client-side

use error types from https://github.com/superroma/testcafe-phoenix/blob/master/src/test-error/type.js,
test run service message commands from https://github.com/superroma/testcafe-phoenix/blob/master/src/runner/test-run/command.js,
worker commands from https://github.com/superroma/testcafe-phoenix/blob/master/src/browser-connection/command.js

Use it for the browser open/close, installations scan. Rename src/browsers to src/browser-connection

We can't do it currently since we need to remove stack frames related to our code (especially, Hammerhead). But this task is complicated by the fact that stack formats differs from browser to browser. However there are some libraries what unifies stack traces.

https://help.github.com/articles/about-protected-branches/

You can filter them before passing to .src()