webdriverio-boneyard / v5 Goto Github PK

The problem

With a Jenkins run with tests recorded as being run on both IE10 and IE11 in the Jenkins report, the Allure plugin-generated report attached to the same Jenkins job only reports the tests as run once, on 'internet explorer'

Moving from graveyard repository:
webdriverio-boneyard/wdio-allure-reporter#131

Environment

• WebdriverIO version: 4.6.2
• Node.js version: 8.9.0
• Standalone mode
• Additional wdio packages used (if applicable): wdio-allure-reporter version 0.7.0.

The problem

In order to avoid regression when adding new features we want to make sure that every code piece is properly unit tested. There are a couple areas where the test coverage is lacking. Let's fix this by adding unit tests for it. Goal is it to have ~95% test coverage across all projects.

The problem

When running something like the following command with spec set it's running all of the specs in the test directory and not just the one specified. In wdio-cli/src/config.js the spec, suite and maybe reporters are set to an array but the code is looking for it to be a string in ConfigParser.js. In v4 they are a string and is split on commas.

Is it intended to have the spec and suite command line arguments as an array now in v5 or should they stay as a string like in v4? I can submit a PR with updates depending on which we use.

./node_modules/.bin/wdio wdio.conf.js --spec ./test/foo.test.js

Environment

• WebdriverIO version: alpha 10
• Node.js version: 8.9.1
• Standalone mode or wdio testrunner: testrunner
• if wdio testrunner, running synchronous or asynchronous tests: sync
• Additional wdio packages used (if applicable):

The wdio-firefox-profile-service package (https://github.com/webdriverio/wdio-firefox-profile-service) needs to be ported to v5.

The problem

The global variable to access the WebdriverIO instance used to be browser within the last versions. This is too specific given that you can automate native mobile applications as well.

Proposed solution

Allow user to also use driver as another option.

From @JulsRicketti on June 29, 2018 19:25

The problem

Can't run my tests after update for wdio-mocha-framework and wdio-spec-reporter were updated to 5.0.0-alpha.7

Environment

• WebdriverIO version: 4.13.1
• Node.js version: 8.9.4
• if wdio testrunner, running synchronous or asynchronous tests: mostly synchronous
• Additional wdio packages used (if applicable):
  wdio-mocha-framework: 5.0.0-alpha.7
  wdio-selenium-standalone-service: 0.0.10
  wdio-spec-reporter: 5.0.0-alpha.7

Details

My tests are run with the following command:

"test:e2e": "node -r dotenv/config ./node_modules/.bin/wdio wdio/wdio.conf.js"

There were no problems previously and if I put the versions back to as they previously were (which is 0.5.13 for wdio-mocha-framework and 0.1.4 for wdio-spec-reporter), the tests run normally.

If I do run with both of the packages on 5.0.0-alpha.7 I get the error message stating that the wdio-spec-reporter module could not be found. If I revert it back to 0.1.4 I will then get an error with a message stating that wdio-mocha-framework couldn't be loaded.

Finally it works as expected once I revert both packages back to the previous versions being used.

Keep in mind that I have double, triple and quadruple checked that the packages have been installed prior to running it.

Link to Selenium/WebdriverIO logs

Full logs for when both packages are at 5.0.0-alpha.7:

node -r dotenv/config ./node_modules/.bin/wdio wdio/wdio.conf.js

local conf not found
/Workspace/MyProject/node_modules/webdriverio/build/lib/launcher.js:163
                        throw _iteratorError;
                        ^

Error: reporter "wdio-spec-reporter" is not installed. Error: Error: Cannot find module 'wdio-spec-reporter'
    at Function.Module._resolveFilename (module.js:538:15)
    at Function.Module._load (module.js:468:25)
    at Module.require (module.js:587:17)
    at require (internal/module.js:11:18)
    at Launcher.initReporters (/Workspace/MyProject/node_modules/webdriverio/build/lib/launcher.js:138:40)
    at new Launcher (/Workspace/MyProject/node_modules/webdriverio/build/lib/launcher.js:73:31)
    at launch (/Workspace/MyProject/node_modules/webdriverio/build/lib/cli.js:452:24)
    at Object.<anonymous> (/Workspace/MyProject/node_modules/webdriverio/build/lib/cli.js:485:5)
    at Module._compile (module.js:643:30)
    at Object.Module._extensions..js (module.js:654:10)
    at Launcher.initReporters (/Workspace/MyProject/node_modules/webdriverio/build/lib/launcher.js:140:35)
    at new Launcher (/Workspace/MyProject/node_modules/webdriverio/build/lib/launcher.js:73:31)
    at launch (/Workspace/MyProject/node_modules/webdriverio/build/lib/cli.js:452:24)
    at Object.<anonymous> (/Workspace/MyProject/node_modules/webdriverio/build/lib/cli.js:485:5)
    at Module._compile (module.js:643:30)
    at Object.Module._extensions..js (module.js:654:10)
    at Module.load (module.js:556:32)
    at tryModuleLoad (module.js:499:12)
    at Function.Module._load (module.js:491:3)
    at Module.require (module.js:587:17)

Full logs for when just wdio-mocha-framework has been updated:

> node -r dotenv/config ./node_modules/.bin/wdio wdio/wdio.conf.js

local conf not found
local conf not found
ERROR: Couldn't load "wdio-mocha-framework" framework. You need to install it with `$ npm install wdio-mocha-framework`!
Error: Cannot find module 'wdio-mocha-framework'
    at Function.Module._resolveFilename (module.js:538:15)
    at Function.Module._load (module.js:468:25)
    at Module.require (module.js:587:17)
    at require (internal/module.js:11:18)
    at Runner.initialiseFramework (/Workspace/MyProject/node_modules/webdriverio/build/lib/runner.js:531:24)
    at Runner._callee$ (/Workspace/MyProject/node_modules/webdriverio/build/lib/runner.js:101:55)
    at tryCatch (/Workspace/MyProject/node_modules/regenerator-runtime/runtime.js:62:40)
    at Generator.invoke [as _invoke] (/Workspace/MyProject/node_modules/regenerator-runtime/runtime.js:296:22)
    at Generator.prototype.(anonymous function) [as next] (/Workspace/MyProject/node_modules/regenerator-runtime/runtime.js:114:21)
    at step (/Workspace/MyProject/node_modules/babel-runtime/helpers/asyncToGenerator.js:17:30)

    at Runner.initialiseFramework (/Workspace/MyProject/node_modules/webdriverio/build/lib/runner.js:537:23)
    at Runner._callee$ (/Workspace/MyProject/node_modules/webdriverio/build/lib/runner.js:101:55)
    at tryCatch (/Workspace/MyProject/node_modules/regenerator-runtime/runtime.js:62:40)
    at Generator.invoke [as _invoke] (/Workspace/MyProject/node_modules/regenerator-runtime/runtime.js:296:22)
    at Generator.prototype.(anonymous function) [as next] (/Workspace/MyProject/node_modules/regenerator-runtime/runtime.js:114:21)
    at step (/Workspace/MyProject/node_modules/babel-runtime/helpers/asyncToGenerator.js:17:30)
    at /Workspace/MyProject/node_modules/babel-runtime/helpers/asyncToGenerator.js:28:13
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)

Copied from original issue: webdriverio/webdriverio#2787

As Wdio involves different contributors, I suggest we use https://github.com/prettier/prettier to keep its codebase always formatted in the same way.

@christian-bromann what do you think?

The problem

Not sure if this is really important but in v4 we had a watch task which was not ported to v5. Let's reevaluate the importance of it and implement it if we see it to be useful.

The problem

Port over the selenium standalone service.

Port the wdio-spec-reporter package and adapt it to the new reporter API.

The problem

Currently multiple methods seem to use different error messages when it comes to invalid / unsupported parameters that are passed.
Let's introduce a generic error that can be used when invalid / unsupported are passed instead so that the error is always the same and logical.

Example:
export const InputError = new Error('InputError, invalid input: number or type of arguments does not match the requirements for this command')

import InputError from './errors'
throw new InputError()

Due to high demand: let's ensure that we handle parallel requests to browser driver gracefully by either having:

• only allowing n number of requests at a time
• or retry requests that have failed that way

I would prefer first approach being implemented in /packages/webdriver/src/request.js

ref: webdriverio/webdriverio#2406

Currently browser driver still have issues fetching shadow DOM elements. Support for this can enabled by a custom element call as implemented in this plugin. Let's bring this functionality into v5 as native support until drivers do fully support it.

Jest is reporting several areas are failing proper unit test coverage. We need to improve coverage so we have a higher confidence level in the quality of the code.

Let's port the wdio-allure-reporter package and adapt it to the new reporter API

The problem

Many users are using WebdriverIO with typescript. We should ensure that these people have a first level support for the new object structure and commands.

Support should be added for:

• webdriver
• webdriverio
• any service that adds commands

The problem

Our cli interface is not allowing to do any inputs if wdio testrunner is running.

The problem

Webdriverio v5 is going to be monorepo project. It will require appropriate changes in GOVERNANCE.md document to handle PR\issues.

e.g.:

WebdriverIO has a large ecosystem of repositories. Some are 'core' in the sense of being central subpackages with a lot of traffic like webdriverio/webdriverio. Others are 'peripheral' in the sense of receiving few changes, being maintained by 1 person or no one, etc..., like webdriverio/cucumber-boilerplate or webdriverio/jasmine-boilerplate. For 'core' repos, committers should not merge their own code straightaway. Like all contributors they should open a PR and get a +1 from another committer. For 'peripheral' repos for which they are the sole maintainer or for which they have a good understanding, it is obtuse to insist on another contributor's +1 and this rule should be relaxed in that case. In sum: use good judgment and don't ram code through without a review when you can help it.

The problem

Stack traces are usually filled with lines that neither belong to WebdriverIO nor to the test spec. Let's make sure we clean the stack trace to only include traces from the test spec and within WebdriverIO.

The problem

I continued to play v5.
I installed

"webdriverio": "5.0.0-alpha.10",
"wdio-mocha-framework": "5.0.0-alpha.10",
"wdio-spec-reporter": "5.0.0-alpha.10"

and when I ran test I got error:

Error: reporter "wdio-spec-reporter" is not installed. Error: Error: Cannot find module 'source-map-support'

repo to reproduce issue - https://github.com/BorisOsipov/wdio-spec-alpha-issue

Environment

• WebdriverIO version: 5.0.0-alpha.10
• Node.js version: v8.11.3
• Standalone mode or wdio testrunner: wdio testrunner
• if wdio testrunner, running synchronous or asynchronous tests: synchronous
• Additional wdio packages used (if applicable):

Link to Selenium/WebdriverIO logs

bo@boubuntu:~/tmp/wdio5$ ./node_modules/.bin/wdio wdio.conf.js
/home/bo/tmp/wdio5/node_modules/webdriver/build/lib/launcher.js:158
                        throw _iteratorError;
                        ^

Error: reporter "wdio-spec-reporter" is not installed. Error: Error: Cannot find module 'source-map-support'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/bo/tmp/wdio5/node_modules/wdio-spec-reporter/build/index.js:7:26)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Launcher.initReporters (/home/bo/tmp/wdio5/node_modules/webdriver/build/lib/launcher.js:135:35)
    at new Launcher (/home/bo/tmp/wdio5/node_modules/webdriver/build/lib/launcher.js:69:31)
    at Object.<anonymous> (/home/bo/tmp/wdio5/node_modules/webdriver/build/lib/cli.js:424:20)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)

Code To Reproduce Issue [ Good To Have ]

Port the wdio-testingbot-service package.

There are a couple of sub packages that haven't been moved to this repo yet. It is not required to move it completely including code and tests but a start would be to have them bootstrapped into this repo.

If you want to move the whole package, create a new issue so everyone is aware that this is taken care of

The problem

If some exception throws in mocha before hooks wdio runner hangs without reporting error.

describe('webdriver.io page', () => {
    before("setup smth", () => {
        browser.url('http://webdriver.io')
        throw new Error("")
    })

    it('should have the right title - the fancy generator way', () => {
        const title = browser.getTitle()
        assert.equal(title, 'WebdriverIO - WebDriver bindings for Node.js')
    })
})

Environment

• WebdriverIO version:5.0.0-alpha.13
• Node.js version:v8.11.3
• Standalone mode or wdio testrunner: wdio testrunner
• if wdio testrunner, running synchronous or asynchronous tests: sync
• Additional wdio packages used (if applicable): spec reporter

Link to Selenium/WebdriverIO logs

 RUNNING  0-0 in chrome - /mocha/mocha.test.js

Test Suites:     0 passed, 1 total (0% completed)
Time:            🕘  16.60s

The Task

Port command:

• selectByAttribute
• selectByIndex
• selectByValue
• selectByVisibleText

to the element scope. Add unit tests (similar to these tests to ensure the right requests are made.

The problem

Port wdio-concise-reporter

A package for local test execution to download, install and start browser drivers.

• Read capabilities from wdio.conf
• Check to see if browser drivers exist in PATH (https://github.com/SeleniumHQ/selenium/blob/master/javascript/node/selenium-webdriver/firefox.js#L409-L417)
  • If yes, start driver
  • If no, download then start driver
• Consumed by launcher.js > run method https://github.com/SeleniumHQ/selenium/blob/master/javascript/node/selenium-webdriver/firefox.js#L409-L417

The problem

Sometimes commands on elements like (click) can't be executed due to the "Element is not clickable at xy" exception. Let's implement a couple of workarounds (explained in this great SO post) as part of the element handler error middleware to prevent users from running into this issue.

Many people want to run tests having a specific set of tags. In v4 there was no tag detection and if you would use the framework functionality it would spawn up a browser and immediately close it.

Let's implement native support for this.

See implementation details here:
https://github.com/webdriverio/webdriverio/blob/master/lib/commands/touchAction.js

The problem

After setting up the repo per https://github.com/webdriverio/v5/blob/master/CONTRIBUTING.md, and making sure the npm run test step is working:

Test Suites: 90 passed, 90 total
Tests: 500 passed, 500 total
Snapshots: 0 total
Time: 86.877s
Ran all test suites.

Try and run an example test like

cd ../wdio5/v5/examples/wdio
npm run test:multiremote

result is:

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @ test:multiremote: `../../packages/wdio-cli/bin/wdio.js ./multiremote/wdio.conf.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the @ test:multiremote script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/pmerwin/.npm/_logs/2018-08-06T19_29_20_273Z-debug.log

Environment

• WebdriverIO version: v5
• Node.js version: 8

Details

Describe in more detail the problem you have been experiencing, if necessary.

Link to Selenium/WebdriverIO logs

0 info it worked if it ends with ok
1 verbose cli [ '/Users/pmerwin/.nvm/versions/node/v8.11.3/bin/node',
1 verbose cli   '/Users/pmerwin/.nvm/versions/node/v8.11.3/bin/npm',
1 verbose cli   'run',
1 verbose cli   'test:multiremote' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'pretest:multiremote',
4 verbose run-script   'test:multiremote',
4 verbose run-script   'posttest:multiremote' ]
5 info lifecycle @~pretest:multiremote: @
6 info lifecycle @~test:multiremote: @
7 verbose lifecycle @~test:multiremote: unsafe-perm in lifecycle true
8 verbose lifecycle @~test:multiremote: PATH: /Users/pmerwin/.nvm/versions/node/v8.11.3/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/Users/pmerwin/Projects/qa/wdio5/v5/examples/wdio/node_modules/.bin:/Users/pmerwin/.nvm/versions/node/v8.11.3/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Users/pmerwin/Library/Android/sdk:/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home:/Users/pmerwin/sc-4.4.11-osx/bin
9 verbose lifecycle @~test:multiremote: CWD: /Users/pmerwin/Projects/qa/wdio5/v5/examples/wdio
10 silly lifecycle @~test:multiremote: Args: [ '-c',
10 silly lifecycle   '../../packages/wdio-cli/bin/wdio.js ./multiremote/wdio.conf.js' ]
11 silly lifecycle @~test:multiremote: Returned: code: 1  signal: null
12 info lifecycle @~test:multiremote: Failed to exec test:multiremote script
13 verbose stack Error: @ test:multiremote: `../../packages/wdio-cli/bin/wdio.js ./multiremote/wdio.conf.js`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/Users/pmerwin/.nvm/versions/node/v8.11.3/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:304:16)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at EventEmitter.emit (events.js:214:7)
13 verbose stack     at ChildProcess.<anonymous> (/Users/pmerwin/.nvm/versions/node/v8.11.3/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at ChildProcess.emit (events.js:214:7)
13 verbose stack     at maybeClose (internal/child_process.js:925:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid @
15 verbose cwd /Users/pmerwin/Projects/qa/wdio5/v5/examples/wdio
16 verbose Darwin 16.7.0
17 verbose argv "/Users/pmerwin/.nvm/versions/node/v8.11.3/bin/node" "/Users/pmerwin/.nvm/versions/node/v8.11.3/bin/npm" "run" "test:multiremote"
18 verbose node v8.11.3
19 verbose npm  v6.2.0
20 error code ELIFECYCLE
21 error errno 1
22 error @ test:multiremote: `../../packages/wdio-cli/bin/wdio.js ./multiremote/wdio.conf.js`
22 error Exit status 1
23 error Failed at the @ test:multiremote script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Code To Reproduce Issue [ Good To Have ]

Not using anything but what is installed by the setup doc:

Sub-packages require a certain setup to be consistent across the project. To ensure this, let's have a script that uses inquirer to ask question regarding type of sub package (core, helper, reporter, runner or service) and name etc.

The task

The debug command is an important command to have an interactive way to fire webdriver commands. We should port this command to the browser scope. Add unit tests.

The problem

Enable to use the debug utilities it should be as simple as passing --inspect to the wdio command and then use the debugger statement. There still seem to be some issues with connecting the runner process correctly to the DevTools app (Node.js v10). This should be fixed before release.

Eventual docs should be updated if something essential is changing.

The problem

Port the wdio-sauce-service.

The problem

We have a lot of packages that want to get released. We should have an easy and straightforward way to do so.

AC:

• figure out how to release individual packages with Lerna
• the process should ensure that the package is compiled and pushed properly
• write a script if necessary
• it should be as simple as running npm run release wdio-cli or npm run release:minor wdio-cli

webdriverio-boneyard/wdio-allure-reporter#133

Questions to answer

1. Do cucumber background steps run before or after wdio hooks? (note that they run before cucumber hooks) (source: background.feature)

For Cucumber we have defined special hooks anyway which are the following:

• beforeFeature
• beforeScenario
• beforeStep
• afterStep
• afterScenario
• afterFeature

I could imagine to add:

• beforeBackground
• afterBackground

2. Is it correct to assume that there is no --exit functionality or that wdio replaces it? (source: exit.feature)

I am not entirely sure what this option in Cucumber is.

3. Is it correct to assume that there is no --fail-fast functionality or that wdio replaces it? (source: fail_fast.feature)

WDIO has a bail option.

4. Does wdio support multiple languages like cucumber? (source: language.feature)

I think it does by applying certain cucumberOpts in the wdio config.

5. Do wdio rerun semantics replace/invalidate the cucumber ones? (source: rerun_formatter.feature)

This is also up to the Cucumber framework. WDIO doesn't do anything specific like this.

6. How does the wdio rerun functionality work with the cucumber rerun functionality (reporting)? (source: step_wrapper_with_options.feature)

It just reruns the step without reporting (besides before/after command reporting). I haven't put much thought into
this as I am not a Cucumber expert.

7. Is strict mode useable with wdio? (source: strict_mode.feature)

What kind of strict mode?

8. Is the high-level summary from cucumber relevant when using wdio? (source: summary_formatter.feature)

I think yes.

9. Is there any special reporting needed for tags functionality? (source: tagged_hooks.feature)

Tags aren't really working with the current integration which I would love to change. Such reporting would
need to be added.

10. Do we need to cope with errors invoking cucumber (e.g. invalid world parameters) ? (source: world_parameters.feature)

Yes.

Decompose the unit test files so there is a 1:1 match with an actual code file. Ex: waitForExist.spec.js would only contain tests for waitForExist.js.

This will provide easier long term maintenance and help reduce the barrier for new contributors

The task

Port the following cookie commands:

• deleteCookies
• getCookies
• setCookies

to the browser scope. Add unit tests (similar to these tests) to ensure the correct requests have been made.

The W3C protocol only accepts certain type of capabilities and throws an error if these are not existing or unknown are added. Let's make sure we handle this gracefully.

The problem

We currently have an own implementation to validate the config that the user is applying to the WebdriverIO session. Let's use jest-validate for this as it is general enough, a well maintained 3rd party tool and does what we want.

This will require to modify the current defined options in the webdriverio and webdriver package.

Port the wdio-mochawesome-reporter

• adapt to new reporter API
• adapt to latest version of mochawesome-report-generator

Need to fix all failing unit tests so that contributors will have confidence in changes they make.

*** Note ***
Update waitUntil with specific error types when ErrorHandler is implemented