wswebcreation / webdriver-image-comparison Goto Github PK

License: MIT License

TypeScript 94.80% JavaScript 5.19% Shell 0.01%

webdriver-image-comparison's Introduction

webdriver-image-comparison

Ultra low-maintenance mode: One or two times a year the library author will spend some time to keep the library useful. Feel free to raise issues and contribute improvements, but please be aware that it may be sometime before a response is given.

This project has been moved to the WebdriverIO core package and is part of the "NEW" Visual Testing packages of WebdriverIO.

webdriver-image-comparison is an image compare module that can be used for different NodeJS Test automation frameworks that support the webdriver protocol.

TO-DO

Make proper documentation
Optimize UT's and increase coverage
Look at the execute methods, maybe they could be optimized

webdriver-image-comparison's People

Contributors

Stargazers

Watchers

webdriver-image-comparison's Issues

Undefined object exception when evaluating versionOffsets.ADDRESS_BAR with iOS13 Safari

Environment (please complete the following information):

Node.js version: v12.3.1
NPM version: 6.9.0
Browser name and version: Safari 13.1
Platform name and version: iPhone 11 XCodeSimulator
Framework name and version: wdio
Plugin name and version: wdio-image-comparison-service 1.6.0
webdriver-image-comparison version: 0.9.1

Config of the automation framework + plugin
A part of my wdio.conf file:

    capabilities:[{
      maxInstances: 1,
      browserName: 'Safari',
      platformName: 'ios',
      'safari:useSimulator': true,
      'safari:deviceType': 'iPhone',
    }],

  services: ['selenium-standalone',
              ['image-comparison', {
                baselineFolder: join(process.cwd(), visualComparisonDirectory, 'baseline'),
                formatImageName: '{tag}-{deviceName}-{width}x{height}',
                screenshotPath: join(process.cwd(), visualComparisonDirectory, 'screenshots'),
                savePerInstance: true,
                autoSaveBaseline: true,
                blockOutStatusBar: true,
                blockOutToolBar: true,
              }],

Describe the bug

When executing a test with a browser.checkScreen command (or any other command which takes screenshots), an exception occurs:

A JavaScript exception occured: undefined is not an object (evaluating 'versionOffsets.ADDRESS_BAR')

To Reproduce
Steps to reproduce the behavior:

Setup XCode and start a iPhone simulator with iOS 13.1
Start a test which contains a browser.checkScreen() command.

Expected behavior

The test shoud take screenshot and compare them like with the previous versions of iOS.

Additional context

After some investigation I realized that there are no OFFSETS defined for iOS13 in file lib/helpers/constants.ts. It seems to work when I manually add values but I can't find the correct real values for this device.

disableCSSAnimation breaks transform-Transformations

Config of the automation framework + plugin
{disableCSSAnimation: true}

Describe the bug
Hi there, thanks for the project. We enabled disableCSSAnimation, so that we don't have to disable animations ourselves. Unfortunately, this caused our page to render incorrectly, so we had to disable it again.

To Reproduce
Steps to reproduce the behavior:

Assume a page that uses transform, for example to center elements.
Set disableCSSAnimation: true.
The transform: none !important; now causes the page to render incorrectly.

Expected behavior
setCustomCss should simply not set transform: none !important; (and the vendor-prefixed variants).

Error: Unsupported image type

Hi, is the element screen shot supported? I have this error each time I run the element comparison.
*Note: it works fine for checkScreen

Node.js version: v8.9.4
NPM version: 6.9.0
Browser name and version: Chrome 74.0.3729.131 (Official Build) (64-bit)
Platform name and version: Windows 10
Framework name and version: Protractor 5.3.1 (protractor-cucumber-framework)
Plugin name and version: protractor-image-comparison 3.1.1

Config of the automation framework + plugin

plugins: [
        {
            // The module name
            package: 'protractor-image-comparison',
            // Some options, see the docs for more
            options: {
                baselineFolder: path.resolve(process.cwd(), './baseline/'),
                formatImageName: `{tag}-{browserName}-{width}x{height}`,
                screenshotPath: path.resolve(process.cwd(), './screenshots/'),
                savePerInstance: true,
                autoSaveBaseline: true,
                debug: true,
                // ... more options
            },
        },
    ],

Describe the bug
AssertionError: expected promise to be fulfilled but it was rejected with 'Error: Unsupported image type'
and the saved images is only 3bytes in size that is only header in there

To Reproduce

expect(await browser.imageComparison.checkElement(element(by.xpath('.//logo')), sourceName, { /* some options*/ })).toEqual(0)

Expected behavior
An image should be saved the same way it is for checkScreen

Log
Though the debug is enabled no real logs are shown in the console for this plugin

not able to make the page scroll with saveFullPageScreen()

Environment:

Node.js version: 11.9.0
NPM version: 6.5.0
Browser name and version: Chrome 75
Platform name and version: Windows 10
webdriver-image-comparison version: 1.5.0

Describe the bug
not able to make the page scroll with saveFullPageScreen() on the following page:
https://www.cpjobs.com/hk/article/hong-kong-international-school-teacher-recruitment-fair

hideAfterFirstScroll leads to distorted full document screenshots

Environment (please complete the following information):

Node.js version: 8.11.4 (but also tries in 10.15.0)
Browser name and version: Chrome 75.0.3770.80
Platform name and version: Debian Jessie
Framework name and version: wdio 5.10.0
Plugin name and version: wdio-image-comparison-service 1.1.0
webdriver-image-comparison version: 0.5.0

Config of the automation framework + plugin
See here: https://github.com/anto-ac/hideAfterFirstScroll-image-comparison-service-issue

Describe the bug
When taking full document screenshot using hideAfterFirstScroll, the generated screenshot has some issues/distortions.

See image (some of the problems are highlighted in black circles):
https://user-images.githubusercontent.com/30859172/59304124-8d478f80-8c8f-11e9-981a-e5b881acacea.png

To Reproduce
Run the test included in this repo:
https://github.com/anto-ac/hideAfterFirstScroll-image-comparison-service-issue

You will need docker and docker-compose to run it.

Expected behavior
The whole page should be captured without distortions.

Log
See here:
https://gist.github.com/anto-ac/0bdf4d51a7049a1a9c9170542cd674bb

Additional context
Add any other context about the problem here.

Is it possible to remove warning message on console?

Hi,

I would like to know if it is possible to not have the console log:

#####################################################################################
 WARNING:
 There was a difference. Saved the difference to
 /home/chk_user/code/test-report/screenshots/diff/desktop_chrome/dois-campos-adicionais-nao-informados-cadastro-de-usuario--1919x1079.png
#####################################################################################

Node.js version: v10.15.3
NPM version: 6.4.1
Browser name and version: Chrome 77.0.3865.75
Platform name and version: Linux
Framework name and version: WebdriverIO 5.13.2
Plugin name and version: wdio-image-comparison-service 1.6.0
webdriver-image-comparison version: 1.6.0

I created a method to validate the image, and try it again with WebdriverIO waitUntil.
So, I don't need the console log on my screen.

visualRegressionElement(element, arquivo) {
        element.waitForDisplayed();
        element.scrollIntoView();
        browser.waitForDisplayedInViewport(element);
        browser.waitUntil(
            () => {
                const diff = browser.checkElement(element, arquivo);
                return diff < 0.15;
            },
            null,
            `Elemento com seletor: '${element.selector}' não está conforme imagem de referência: ${arquivo}`,
        );
    },

In Typescript browser.imageComparison is undefined

I have a repo that is in typescript and I am trying to use this image comparison but I get the error
TypeError: Cannot read property 'checkScreen' of undefined

here is the code that I thought would work that is causing the error

expect(await browser.imageComparison.checkScreen('fullpage', {ignoreColors: true, ignoreAntialiasing: true})).to.be.below(.05);
let visualPercentage = await browser.imageComparison.checkScreen('fullpage', {ignoreColors: true, ignoreAntialiasing: true})

Environment:

Node.js version: v14.20.0
NPM version: 6.14.17
Browser name and version: Chrome 104
Platform name and version: Windows 11
Framework name and version: Webdriverio
Plugin name and version: webdriver-image-comparison
webdriver-image-comparison version: 0.22.0

Can't take a screenshot with Appium (browser.checkElement is not a function)

Node.js version: 15.0.0
NPM version: 7.0.2
Browser name and version: Chrome83
Platform name and version: Windows 10
WebdriverIO version: 7.16.14
wdio-image-comparison-service version: 3.1.0
Appium 2.0.0-beta.25 ( 1.22.2)
TS version: 10.4.0
Config of WebdriverIO + wdio-image-comparison-service

    services: [
        ['appium'],
        ['chromedriver'],
        ['image-comparison',
        // The options
        {
            // Some options, see the docs for more
            baselineFolder: join(process.cwd(), './test/visual test/baseline/'),
            formatImageName: '{tag}-{logName}-{width}x{height}',
            screenshotPath: join(process.cwd(), './test/visual test/chekc/'),
            // savePerInstance: true,
            autoSaveBaseline: true,
            // blockOutStatusBar: true,
            // blockOutToolBar: true,
            isHybridApp:true
            // clearRuntimeFolder: true
            // Without this option, it errors out only when mismatch is above 1.23%(by default)
            // ... more options
        }],  
    ],

Describe the bug

after launch, Chrome opens on Pixel 3 (Android Studio emulator).
URL is being navigated
Finds a layout element.
await browser.checkElement(await element, 'simple', {}) is called
Error appears in console "browser.checkElement is not a function"
Playable for any other image comparison or image save function

Repeat for any other image comparison function or image save function

To Reproduce
start Android Emulator Pixel 3
npm i
npm test

https://github.com/Searcher865/wdio-with-appium-example-

Expected behavior
Test completed successfully. A screenshot of the element is taken and saved.

2022-02-01T14:28:12.563Z INFO @wdio/cli:launcher: Run onPrepare hook
2022-02-01T14:28:12.570Z INFO chromedriver: Start Chromedriver (C:\automation\wdio-with-appium-example-\node_modules\chromedriver\lib\chromedriver\chromedriver.exe) with args --port=9515 --url-base=/
2022-02-01T14:28:12.597Z INFO chromedriver: Starting ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416}) on port 9515
2022-02-01T14:28:12.598Z INFO chromedriver: Only local connections are allowed.
2022-02-01T14:28:12.598Z INFO chromedriver: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2022-02-01T14:28:12.599Z INFO chromedriver: ChromeDriver was started successfully.
2022-02-01T14:28:16.060Z WARN @wdio/config:ConfigParser: pattern ./test/specs/check_form.ts did not match any file
2022-02-01T14:28:16.061Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2022-02-01T14:28:16.062Z INFO @wdio/local-runner: Start worker 0-0 with arg: run,./wdio.conf.js
[0-0] 2022-02-01T14:28:16.639Z INFO @wdio/local-runner: Run worker command: run
[0-0] RUNNING in Android - C:\automation\wdio-with-appium-example-\test\specs\simple.ts
[0-0] 2022-02-01T14:28:20.973Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-0] 2022-02-01T14:28:21.038Z INFO webdriver: [POST] http://localhost:4723/session
[0-0] 2022-02-01T14:28:21.039Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: {
[0-0]       platformName: 'Android',
[0-0]       'appium:deviceName': 'Pixel 3',
[0-0]       'appium:platformVersion': '11.0',
[0-0]       'appium:automationName': 'UIAutomator2',
[0-0]       'appium:browserName': 'Chrome',
[0-0]       'appium:chromedriverExecutable': 'C:/WebDriver/bin/chromedriver.exe',
[0-0]       'appium:appiumVersion': '2.0.0-beta.25',
[0-0]       'appium:waitforTimeout': '30000',
[0-0]       'appium:commandTimeout': '30000'
[0-0]     },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: {
[0-0]     platformName: 'Android',
[0-0]     'appium:deviceName': 'Pixel 3',
[0-0]     'appium:platformVersion': '11.0',
[0-0]     'appium:automationName': 'UIAutomator2',
[0-0]     'appium:browserName': 'Chrome',
[0-0]     'appium:chromedriverExecutable': 'C:/WebDriver/bin/chromedriver.exe',
[0-0]     'appium:appiumVersion': '2.0.0-beta.25',
[0-0]     'appium:waitforTimeout': '30000',
[0-0]     'appium:commandTimeout': '30000'
[0-0]   }
[0-0] }
[0-0] 2022-02-01T14:28:28.251Z INFO wdio-image-comparison-service: Adding commands to Multi Browser:  [
[0-0]   'maxInstances',
[0-0]   'platformName',
[0-0]   'appium:deviceName',
[0-0]   'appium:platformVersion',
[0-0]   'appium:automationName',
[0-0]   'appium:browserName',
[0-0]   'appium:chromedriverExecutable',
[0-0]   'appium:appiumVersion',
[0-0]   'appium:waitforTimeout',
[0-0]   'appium:commandTimeout',
[0-0]   'protocol',
[0-0]   'hostname',
[0-0]   'port',
[0-0]   'path'
[0-0] ]
[0-0] 2022-02-01T14:28:28.254Z ERROR @wdio/utils:shim: TypeError: Cannot read property 'browserName' of undefined
[0-0]     at getInstanceData (C:\automation\wdio-with-appium-example-\node_modules\wdio-image-comparison-service\build\utils.js:45:37)
[0-0]     at WdioImageComparisonService.addCommandsToBrowser (C:\automation\wdio-with-appium-example-\node_modules\wdio-image-comparison-service\build\service.js:86:53)
[0-0]     at WdioImageComparisonService.before (C:\automation\wdio-with-appium-example-\node_modules\wdio-image-comparison-service\build\service.js:67:14)
[0-0]     at C:\automation\wdio-with-appium-example-\node_modules\@wdio\utils\build\shim.js:88:27
[0-0]     at new Promise (<anonymous>)
[0-0]     at C:\automation\wdio-with-appium-example-\node_modules\@wdio\utils\build\shim.js:85:47
[0-0]     at Array.map (<anonymous>)
[0-0]     at executeHooksWithArgsShim (C:\automation\wdio-with-appium-example-\node_modules\@wdio\utils\build\shim.js:85:33)
[0-0]     at Runner.run (C:\automation\wdio-with-appium-example-\node_modules\@wdio\runner\build\index.js:99:48)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:93:5)
[0-0] 2022-02-01T14:28:28.266Z INFO webdriver: COMMAND navigateTo("https://webdriver.io/")
[0-0] 2022-02-01T14:28:28.266Z INFO webdriver: [POST] http://localhost:4723/session/1a9fb82b-c859-4faf-8294-0abd463c79f4/url
[0-0] 2022-02-01T14:28:28.266Z INFO webdriver: DATA { url: 'https://webdriver.io/' }
[0-0] 2022-02-01T14:28:32.617Z INFO webdriver: COMMAND findElement("css selector", ".hero__title")
[0-0] 2022-02-01T14:28:32.618Z INFO webdriver: [POST] http://localhost:4723/session/1a9fb82b-c859-4faf-8294-0abd463c79f4/element
[0-0] 2022-02-01T14:28:32.618Z INFO webdriver: DATA { using: 'css selector', value: '.hero__title' }
[0-0] 2022-02-01T14:28:32.685Z INFO webdriver: RESULT {
[0-0]   ELEMENT: 'c6f63576-45f4-4e92-825f-2a4967a78468',
[0-0]   'element-6066-11e4-a52e-4f735466cecf': 'c6f63576-45f4-4e92-825f-2a4967a78468'
[0-0] }
[0-0] TypeError in "Test one.Simple test"
TypeError: browser.checkElement is not a function
    at C:\automation\wdio-with-appium-example-\test\specs\simple.ts:18:23
    at Generator.next (<anonymous>)
    at fulfilled (C:\automation\wdio-with-appium-example-\test\specs\simple.ts:4:58)
    at processTicksAndRejections (node:internal/process/task_queues:93:5)
[0-0] 2022-02-01T14:28:32.722Z INFO webdriver: COMMAND deleteSession()
[0-0] 2022-02-01T14:28:32.723Z INFO webdriver: [DELETE] http://localhost:4723/session/1a9fb82b-c859-4faf-8294-0abd463c79f4
[0-0] FAILED in Android - C:\automation\wdio-with-appium-example-\test\specs\simple.ts
2022-02-01T14:28:33.247Z INFO @wdio/cli:launcher: Run onComplete hook
Report successfully generated to allure-report
Allure report successfully generated

I would be very grateful for any help.

Option to hide elements

Hey there, thanks for your work on this. I'm excited at the prospect of getting visual regression back into wdio v5.

Sometimes I want to capture an element while hiding other elements that may exist within or on top of it. The location and size of the elements I want to hide can change, so I'd like to approach it more programmatically.

The previous wdio-visual-regression service had a "hide" option when checking that I presume would set CSS display: none on the specified elements prior to capturing. It seems that might be useful here.

If there's a way to achieve this that I missed or if this request is better suited to wdio-image-comparison-service, I can move it over there. Thanks for your consideration!

check* functions not compatible with newer iPad devices

Environment (please complete the following information):

Node.js version: v13.13.0
NPM version: 6.14.4
Browser name and version: iPad 7th
Platform name and version: osVersion 13
Framework name and version: WDIO 6.0.13
Plugin name and version: wdio-image-comparison-service 1.13.0
webdriver-image-comparison version: 0.14.0

Config of the automation framework + plugin
Testing on BrowserStack - capability looks like this:
capabilities: [ { 'bstack:options' : { "osVersion" : "13", "deviceName" : "iPad 7th", "realMobile" : "true", "local" : "false", }, "browserName" : "[iOS] iPad 7th", } ],

Describe the bug
calling check* (e.g. checkScreen()) functions fail due to this library not being able to determine the majorVersion in the getIosStatusAddressToolBarHeight function

To Reproduce
execute any check* function with the wdio-image-comparison-service for a newer ipad device (ipad pro 2nd or ipad 7th) with browser stack.

Expected behavior
the check* functions should not fail.

Log

[iPad 7th MAC 12.1 #0-0] An unknown server-side error occurred while processing the command. Original error: null is not an object (evaluating 'match[1]')
[iPad 7th MAC 12.1 #0-0] unknown error: An unknown server-side error occurred while processing the command. Original error: null is not an object (evaluating 'match[1]')
[iPad 7th MAC 12.1 #0-0] at processTicksAndRejections (internal/process/task_queues.js:97:5)
[iPad 7th MAC 12.1 #0-0] at Object. (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/methods/screenshots.js:86:32)
[iPad 7th MAC 12.1 #0-0] at step (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/methods/screenshots.js:44:23)
[iPad 7th MAC 12.1 #0-0] at Object.next (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/methods/screenshots.js:25:53)
[iPad 7th MAC 12.1 #0-0] at /Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/methods/screenshots.js:19:71
[iPad 7th MAC 12.1 #0-0] at Object.getBase64FullPageScreenshotsData (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/methods/screenshots.js:60:12)
[iPad 7th MAC 12.1 #0-0] at Object. (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/commands/saveFullPageScreen.js:87:46)
[iPad 7th MAC 12.1 #0-0] at step (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/commands/saveFullPageScreen.js:33:23)
[iPad 7th MAC 12.1 #0-0] at Object.next (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/commands/saveFullPageScreen.js:14:53)
[iPad 7th MAC 12.1 #0-0] at fulfilled (/Users/sel-sayed/work/automation-testing/node_modules/webdriver-image-comparison/build/commands/saveFullPageScreen.js:5:58)
[iPad 7th MAC 12.1 #0-0] at Browser.next [as checkFullPageScreen] (/Users/sel-sayed/work/automation-testing/node_modules/webdriver/node_modules/@wdio/utils/build/monad.js:116:27)
[iPad 7th MAC 12.1 #0-0] at World. (/Users/sel-sayed/work/automation-testing/automation/step_definitions/step.js:21:18)

Additional context
The problem seems to be in this line here:

webdriver-image-comparison/lib/clientSideScripts/getIosStatusAddressToolBarHeight.ts

Line 9 in 9ea2e22

const match = (navigator.appVersion).match(/OS (\d+)_(\d+)_?(\d+)?/);

The regex fails for newer ipad devices and the subsequent line which extracts the major version fails. If I modify the major version to 13 then the library works as expected. I am unsure how to retrieve the values of navigator.appVersion to re-write the regex as a solution myself.

postponed image saved in workspace (mobile)

WebdriverIO version: 5.7.15
Mode: WDIO Testrunner
If WDIO Testrunner, running sync/async: sync
Node.js version: 10.15.1
NPM version: 6.4.1
Platform name and version: Browserstack
{
"os_version": "9.0",
"device": "Google Pixel 3",
"real_mobile": "true",
"browserstack.appium_version": "1.12.1"
}
Additional wdio packages used (if applicable): "@wdio/browserstack-service": "^5.7.13", "wdio-image-comparison-service": "^1.1.0"
wdio.conf
services: [["image-comparison",
{
baselineFolder: join(process.cwd(), "./visualRegressionReferenceImages/"),
formatImageName: "{logName}-{tag}-{width}x{height}",
screenshotPath: join(process.cwd(), ".tmp/"),
clearRuntimeFolder: true,
savePerInstance: true,
autoSaveBaseline: true,
blockOutStatusBar: true,
blockOutToolBar: true
}], "browserstack"],

Describe the bug

Visual Regression Test:
expectation:
I want to scroll to an element and save the layout of the element as an image.

Problem only on Mobile:
The image is stored incorrectly. Browserstack indicates that the image was properly saved, but a postponed image is saved in the workspace.
Problem only occurs when several ITs running. If only one IT is running then it will work.

Element:".prod-info__box-container.js-prod-info-box"
URL: https://www.mytoys.de/birkmann-kekskunst-zum-selbermachen-4283014.html

Image Workspace

Image Browserstack

approach:
-browser.pause(3000), before save Picture
-Instead of searching at the search bar, we went directly to the specific URL.

To Reproduce

element.scrollIntoView();
browser.saveElement(element, imageNameMobile, {});

browser.checkElement(element, imageNameMobile, {});

Full Page scrolling doesn't work out of the box for all websites

The project now used this method to automatically scroll.

This is now being done mostly on the HTML element, but there are sites that only scroll on the body element. We need to make the above mentioned method smarter.
Here are two websites to use for testing

scroll on body: https://www.insightinvestment.com/cookie-policy/
scroll on html and not on body: https://wswebcreation.github.io/protractor-image-comparison/

There is a way to test this

// this was for https://www.insightinvestment.com/cookie-policy/
const htmlNode = document.querySelector('html')
htmlNode.scrollHeight
// 936
htmlNode.clientHeight
// 936
const bodyNode = document.querySelector('body')
bodyNode.scrollHeight
// 7557
bodyNode.clientHeight
// 936

I could determine for the html or body which element needs to be used for scrolling just a simple compare bigger scrollHeight vs clientHeight

hideScrollbars doesn't work in mobile emulation

Hi!

I'm trying to make and compare screenshots on the mobile version of my site, but I have noticed that scrollbars don't hide despite hideScrollbars parameter is set to true.

There is project example with result screenshot:
https://github.com/vmrgvn/image-comparison

Please, help me to find out the problem.
Thanks!

Environment (please complete the following information):

Node.js version: v15.12.0
NPM version: 7.6.3
Browser name and version: Chrome 9
Platform name and version: macOS (Catalina)

To Reproduce
Use the project configuration above

Expose browser version as a variable for formatImageName

Is your feature request related to a problem? Please describe.
Image rendering can vary between browser versions. It would be nice to have a direct way of including the browser version in the image name similar to the wdio4 vr service.

Describe the solution you'd like
To expose a {browserVersion} var that can be picked up by formatImageName option

Describe alternatives you've considered
browser.capabilities.version

saveScreen/checkScreen always scroll to the top

Environment (please complete the following information):

Node.js version: [e.g. v13.10.1]
NPM version: [e.g. 6.14.2]
Browser name and version: [e.g. Chrome 80]
Platform name and version: [e.g. Windows 10]
Framework name and version: [e.g. WDIO V5]
webdriver-image-comparison version: [e.g. 1.12.2]

Describe the bug
Scrolling using $(...).scrollIntoView(), than using saveScreen/checkScreen leads to a scroll back to the top(y=0).

To Reproduce

Scroll to the bottom using a seletor which is usually at the bottom: $('footer').scrollIntoView()
Take a screenshot using saveScreen/checkScreen

Expected behavior
No scroll back to the top or a screenshot of the area which I scrolled to

Sourcemaps require lib files

The sourcemaps of webdriver-image-comparison link to the source file, e.g. build/index.js.map is

{"version":3,"file":"index.js","sourceRoot":"","sources":["../lib/index.ts"],"names":[],"mappings":";;AAAA,+BAA+B;AAW7B,oBAXK,cAAS,CAWL;AAVX,oDAA+C;AAa7C,qBAbK,oBAAU,CAaL;AAZZ,sDAAiD;AAa/C,sBAbK,qBAAW,CAaL;AAZb,oEAA+D;AAa7D,6BAbK,4BAAkB,CAaL;AAZpB,sDAAiD;AAa/C,sBAbK,qBAAW,CAaL;AAZb,wDAAmD;AAajD,uBAbK,sBAAY,CAaL;AAZd,sEAAiE;AAa/D,8BAbK,6BAAmB,CAaL"}

Since the lib files are not included in the NPM package this makes debugging of webdriver-image-comparison difficult.

'saveScreen' method does not work on pages with '*/xml' content type

Environment (please complete the following information):

Node.js version: [10.16.0]
NPM version: [6.9.0]
Browser name and version: [Firefox, IE, Edge]
Platform name and version: [Windows 10, Mac OS X]
Framework name and version: [@wdio/jasmine-framework 5.11.0 ]
Plugin name and version: [wdio-image-comparison-service 1.3.0]
webdriver-image-comparison version: [0.6.0]

Config of the automation framework + plugin

services: [
    ['image-comparison',
      {
        baselineFolder: join(process.cwd(), './tools/jasmine/image/expected/'),
        formatImageName: '{tag}-{width}x{height}',
        screenshotPath: join(process.cwd(), './tools/jasmine/image/actual'),
        savePerInstance: true,
        autoSaveBaseline: true,
        blockOutStatusBar: true,
        blockOutToolBar: true,
        returnAllCompareData: true,
        clearRuntimeFolder: true
      }
    ]
  ]

Describe the bug
In case of visit any page with content type 'text/xml', 'application/xml', 'application/xhtml', 'saveScreen' method can't make screenshot on page due to different reasons:

Firefox 68 - javascript error: TypeError: document.body is null
IE 11- Error: JavaScript error
Edge 17 - Error: Unable to get property 'style' of undefined or null reference

However, it correctly works on Chrome 75 and Safari 12.

To Reproduce

describe('XML tests', () => {
    it('XML is correctly displayed for content type text xml', () => {
      browser.url('put here any site with text/xml content type');
      browser.pause(3000);
      browser.saveScreen('test');
  });
});

Expected behavior
Screenshot should be taken.

Log
Example for Firefox:

2019-07-29T16:29:32.757Z INFO webdriver: COMMAND navigateTo("testPage.xml")
2019-07-29T16:29:32.758Z INFO webdriver: [POST] http://127.0.0.1:4444/wd/hub/session/aeae0af7-b890-df49-83b1-5c8b07a1e3ca/url
2019-07-29T16:29:32.758Z INFO webdriver: DATA { url:
   'testPage.xml' }
2019-07-29T16:29:35.942Z INFO webdriver: COMMAND saveScreen("test")
2019-07-29T16:29:35.944Z INFO webdriver: COMMAND executeScript("return (function hideScrollBars(hide) {
    if (hide) {
        document.body.style.overflow = 'hidden';
    }
    else {
        document.body.style.overflow = '';
    }
}).apply(null, arguments)", <object>)
2019-07-29T16:29:35.945Z INFO webdriver: [POST] http://127.0.0.1:4444/wd/hub/session/aeae0af7-b890-df49-83b1-5c8b07a1e3ca/execute/sync
2019-07-29T16:29:35.945Z INFO webdriver: DATA { script:
   'return (function hideScrollBars(hide) {\n    if (hide) {\n        document.body.style.overflow = \'hidden\';\n    }\n    else {\n        document.body.style.overflow = \'\';\n    }\n}).apply(null, arguments)',
  args: [ true ] }
2019-07-29T16:29:35.951Z DEBUG webdriver: request failed due to response error: javascript error
2019-07-29T16:29:35.951Z WARN webdriver: Request failed due to TypeError: document.body is null

HTML report for viewing the baseline and actual image highlighting the difference

Expecting a similar html report generated by jest-screenshot.
jest-screenshot-report.zip

Plugin not working against native mobile android application

Environment (please complete the following information):

Node.js version: 8.12
NPM version: 6.4.1

Capabilities

 { automationName: 'UiAutomator2',
        Application: 'kitchensinkapp',
        deviceName: 'Android Emulator',
        avd: 'Pixel_2_API_28',
        platformVersion: '9.0',
        platformName: 'ANDROID',
        appPackage: 'com.kitchensinkapp',
        appActivity: 'com.kitchensinkapp.MainActivity' 
}

Config of the automation framework + plugin
This was in use in a WDIO test using the comparison plugin.

Use case
We call saveFullPageScreen()

await browser.saveFullPageScreen('fullPage', { /* some options*/ });

which results in this error.

2019-06-06T21:52:46.068Z INFO webdriver: COMMAND saveFullPageScreen("fullPage", <object>)
2019-06-06T21:52:46.074Z INFO webdriver: COMMAND executeScript("return (function hideScrollBars(hide) {
    if (hide) {
        document.body.style.overflow = 'hidden';
    }
    else {
        document.body.style.overflow = '';
    }
}).apply(null, arguments)", <object>)
2019-06-06T21:52:46.074Z INFO webdriver: [POST] http://10.10.29.235:4000/wd/hub/session/fafbde0f-f066-41b9-89c8-67806a13f9ff/execute/sync
2019-06-06T21:52:46.074Z INFO webdriver: DATA { script:
   'return (function hideScrollBars(hide) {\n    if (hide) {\n        document.body.style.overflow = \'hidden\';\n    }\n    else {\n        document.body.style.overflow = \'\';\n    }\n}).apply(null, arguments)',
  args: [ false ] }
2019-06-06T21:52:46.079Z DEBUG webdriver: request failed due to response error: unknown error
2019-06-06T21:52:46.080Z WARN webdriver: Request failed due to Method is not implemented
2019-06-06T21:52:46.080Z INFO webdriver: Retrying 1/3

saveFullPageScreen() comes from https://github.com/wswebcreation/wdio-image-comparison-service/blob/master/lib/index.js#L65
Which in turn calls saveFullPageScreen() from https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/commands/saveFullPageScreen.ts#L16
This error specifically is thrown from beforeScreenshot() which is in https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/helpers/beforeScreenshot.ts#L13

The capabilities:

2019-06-06T21:52:34.420Z INFO webdriver: DATA { capabilities:
   { alwaysMatch:
      { automationName: 'UiAutomator2',
        Application: 'kitchensinkapp',
        deviceName: 'Android Emulator',
        avd: 'Pixel_2_API_28',
        platformVersion: '9.0',
        platformName: 'ANDROID',
        appPackage: 'com.kitchensinkapp',
        appActivity: 'com.kitchensinkapp.MainActivity' },
     firstMatch: [ {} ] },
  desiredCapabilities:
   { automationName: 'UiAutomator2',
     Application: 'kitchensinkapp',
     deviceName: 'Android Emulator',
     avd: 'Pixel_2_API_28',
     platformVersion: '9.0',
     platformName: 'ANDROID',
     appPackage: 'com.kitchensinkapp',
     appActivity: 'com.kitchensinkapp.MainActivity' } }

Code calling the saveFullPageScreen()

describe('Do the thing', () => {
    it('should do the thing', async () => {
        
		... do some button clicks...
		
        await browser.saveFullPageScreen('fullPage', { /* some options*/ });

    });
});

Basically my question is does this plugin support mobile native apps? I thought using browser.execute() from WDIO would ensure the command would get mapped to Appium protocol.

It is news to me that Appium apparently does not implement the :sessionId/execute/sync endpoint?

Invalid Chai property: toEqual. Did you mean "equal"?

Environment (please complete the following information):

Node.js version: v13.12.0
NPM version: 6.14.4
Browser name and version: Chrome 80.0.3987.149
Platform name and version: MacOS Mojave 10.14.6
Framework name and version: [e.g. Protractor 5.1.0]
WebdriverIO version: [e.g. 5.1.0]
Plugin name and version: wdio-image-comparison-service version: "^1.12.2"
"webdriver-image-comparison": "^0.13.2"

Config of the automation framework + plugin
WDIO, Docker, WDIO Image comparison

Describe the bug
The baseline is created. While running test

To Reproduce

        it('Navigate to create new environment', () => {
            leftMenuPageComp.dblClick_mnuEnvs();
            browser.saveFullPageScreen('_EnvScreen', {  });
            expect(browser.checkFullPageScreen('_EnvScreen', {  })).toEqual(0);

            leftSnackMenuPageComp.click_mnuCreateEnv();

[Include code or an example repository that can easily be set up]

Expected behavior
Test suppose to pass or show the difference

Log

➜  wdio git:(poc-wdio-image-comparison-service) ✗ make run-chrome suite=smoke.basic.js
docker-compose -f docker/docker-compose.yml exec -T test_env_chrome bash -c "./node_modules/.bin/wdio wdio.conf.chrome.local.js --baseUrl=https://******** --spec smoke.basic.js"

Execution of 1 spec files started at 2020-04-14T01:23:57.999Z

[0-0] RUNNING in chrome - /test/specs/smoke.basic.js
[0-0]  SEE THIS -> Run id: 79037693
[0-0] Error in "Platform - Create user and role full scenario Create env  Navigate to create new env"
Invalid Chai property: toEqual. Did you mean "equal"?
[0-0] Error in "Platform - Create user and role full scenario Create env  Feed in env details and submit"
element ("#envForm_name") still not existing after 90000ms
[0-0] Error in "Platform - Create user and role full scenario Create env  Check env is visible in the list of envs"
element ("//input[@placeholder='Filter ...']") still not clickable after 90000ms
[0-0] FAILED in chrome - /test/specs/smoke.basic.js

 "dot" Reporter:
..FFF

 "spec" Reporter:
------------------------------------------------------------------
[chrome  linux #0-0] Spec: /workspace/test/specs/smoke.basic.js
[chrome  linux #0-0] Running: chrome on linux
[chrome  linux #0-0] Session ID: 4d3b127257bdaf965051b15e715b8976
[chrome  linux #0-0]
[chrome  linux #0-0] Platform - Create user and **** full scenario
[chrome  linux #0-0]     Login
[chrome  linux #0-0]        ✓ Login to *****
[chrome  linux #0-0]
[chrome  linux #0-0]     Create env
[chrome  linux #0-0]        ✓ Top menu go to services section
[chrome  linux #0-0]        ✖ Navigate to create new env
[chrome  linux #0-0]        ✖ Feed in env details and submit
[chrome  linux #0-0]        ✖ Check env is visible in the list of envs
[chrome  linux #0-0]
[chrome  linux #0-0] 2 passing (3m 13.8s)
[chrome  linux #0-0] 3 failing
[chrome  linux #0-0]
[chrome  linux #0-0] 1) Create env  Navigate to create new env
[chrome  linux #0-0] Invalid Chai property: toEqual. Did you mean "equal"?
[chrome  linux #0-0] Error: Invalid Chai property: toEqual. Did you mean "equal"?
[chrome  linux #0-0]     at Object.proxyGetter [as get] (/workspace/node_modules/chai/lib/chai/utils/proxify.js:75:17)
[chrome  linux #0-0]     at Context.<anonymous> (/workspace/test/pages/_test-modules/services.env.module.js:20:68)
[chrome  linux #0-0]

Additional context

iPhone 11 Pro and Max innerHeight is not correct

Version 0.10.0 now supports iOS 13, but there is a strange issue with the iPhone 11 Pro and Max. The innerHeight that is given back is 5 pixels smaller than is should be making the toolbar blockout not correct. It should be 640, but is is giving back 635.

I still need to figure out how to fix this, see the image below

Remove console output "ignoredBoxes"

Environment (please complete the following information):

webdriver-image-comparison version: [e.g. 1.0.0]
0.4.5

Console log is flooded with "ignoredBoxes" created by lib/methods/images.ts line 135.
Please remove this console output:
console.log('ignoredBoxes = ', ignoredBoxes);

'Cannot read property 'options' of undefined' when calling `checkScreen`

Environment (please complete the following information):

Node.js version: 10.15.3
NPM version: 6.9.0
Browser name and version: Chrome 73
Platform name and version: macOS 10.14
Framework name and version: WebdriverIO 5.7.2
Plugin name and version: homemade
webdriver-image-comparison version: 0.1.0

Config of the automation framework + plugin

config.framework = 'jasmine';
config.capabilities = [
  {
    maxInstances: 5,
    browserName: 'chrome',
    'goog:chromeOptions': {
        args: []
    }
  },
];

Describe the bug
When I call checkScreen in the middle of my test with WebdriverIO I get the error:
Cannot read property 'options' of undefined

I don't get a stracktrace for where the error happens, but it happens in the call to checkScreen.

To Reproduce
This is not very beautiful, but a quick and dirty way to get the values for making the call to checkScreen.

import { BaseClass, checkScreen } from 'webdriver-image-comparison';
import { Methods } from 'webdriver-image-comparison/build/methods/methods.interface';
import { InstanceData } from 'webdriver-image-comparison/build/methods/instanceData.interfaces';
import { CheckScreenOptions } from 'webdriver-image-comparison/build/commands/screen.interfaces';
import { ImageCompareResult } from 'webdriver-image-comparison/build/methods/images.interfaces';
import { ComparisonResult } from '../screenshots/comparison-result';

export async function webCompareScreen(tag): Promise<ComparisonResult> {
  const base = new BaseClass({});
  const methods: Methods = {
    executor: browser.executeScript,
    screenShot: browser.takeScreenshot,
  };
  const instanceData = getInstanceData();
  const folders = base.folders;
  const checkScreenOptions: CheckScreenOptions = {
    wic: base.defaultOptions,
    method: {},
  };
  const imageCompareResult = (await checkScreen(
    methods,
    instanceData,
    folders,
    tag,
    checkScreenOptions
  )) as ImageCompareResult;
  return imageCompareResult as ComparisonResult;
}

function getInstanceData(): InstanceData {
  // @ts-ignore Driver is given magically by Webdriverio
  const capabilities = driver.capabilities;
  return {
    browserName: capabilities.browserName,
    deviceName: '',
    logName: '',
    name: '',
    nativeWebScreenshot: false,
    platformName: '',
  };
}

Expected behavior
I expected to get an ImageCompareResult.

Log

[chrome #0-0] TypeError: Cannot read property 'options' of undefined
[chrome #0-0]     at <Jasmine>
[chrome #0-0]     at <Jasmine>

Offset in IE11 for checkElement

Environment (please complete the following information):

Node.js version: [e.g. 8.9.1] 8.11.3
NPM version: [e.g. 5.8.0] 5.6.0
Browser name and version: [e.g. Chrome 68] IE11
Platform name and version: [e.g. Windows 10] Windows 8.1
Framework name and version: [e.g. Protractor 5.1.0] Protractor
Plugin name and version: [e.g. protractor-image-comparison 3.0.0] protractor-image-comparison 3.1.0
webdriver-image-comparison version: [e.g. 1.0.0]

Config of the automation framework + plugin
An example of how you configured the protractor-image-comparison

Describe the bug
CheckElement function on IE11 is not taking snapshot of the element completely. There is a slight deviation in the offset.
It is working fine in Chrome, Safari.

To Reproduce
Steps to reproduce the behavior:
Perform checkElement on IE11.

Expected behavior
Should take complete snapshot of the element.

Log
If applicable, add logs to help explain your problem. If you don't have log, enable 'debug:true' in the config and paste the log here.
Please use proper markdown to style it

Additional context
Add any other context about the problem here.

How to test multiple viewports?

I think that a suggested / recommended strategy for testing multiple viewports should be documented.

For example, are individual tests required to explicitly set the viewport dimensions prior to capturing a screenshot, or is there a more global option available that can assist in setting up global viewport dimensions?

Great job on this by the way!

Tool bar and status bar not ignored on mobile android devices

Environment:

Node.js version: v14.15.0
NPM version: 7.19.0
Browser name and version: Chrome
Platform name and version: Android 9
Framework name and version: webdriverio v7
webdriver-image-comparison version: 3.0.0

Config of the automation framework + plugin
Using Lambdatest https://www.lambdatest.com/ to use mobile devices and device capability is
{
build: "Test Run",
name : "Test Galaxy S10 Android 9",
browserName : "Chrome",
platform : "Android",
deviceName : "Galaxy S10",
platformVersion : "9"
}

Describe the bug
Method saveFullPageScreen does not remove/ignore toolbar and statusbar in chrome android 9 devices, see screenshot

To Reproduce
Steps to reproduce the behavior:

 browser.url('https://preactjs.com/');
        browser.saveFullPageScreen(imageName, {
            fullPageScrollTimeout: 3000,
        });

Expected behavior
toolbar and status bar on chrome android 9 devices is ignored/not visible in full page screenshot

Hi, I'm trying to take and compare full page screenshots on android devices. I'm not using saucelabs or browserstack, I'm using lambdatest (mentioned above).
On iOS devices it's all fine, but on android devices the toolbar and statusbar are not irgnored. blockOutStatusBar: true, blockOutToolBar: true are set.

Can you please help?

The percentage difference is rounded and passes the tests

Environment (please complete the following information):

Node.js version: v16.14.0
NPM version: 8.3.1
Browser name and version: Chrome 100
Platform name and version: MacOS 12.3.1, 4.14.243-185.433.amzn2.x86_64 and 4.14.219-164.354.amzn2.x86_64
Framework name and version: WebdriverIO 2.32.3
Plugin name and version: serenity-js/webdriverio 2.32.3
webdriver-image-comparison version: 3.1.0

Describe the bug
If you use the default configuration, the value of rawMisMatchPercentage is false.

If you have an image that fails by a mistmatch of 0.001%, it will be rounded to 0.00% and the service will return that the two images are the same (consequently, you can enter visual failures and not see them because they are so minuscule).

To Reproduce
Steps to reproduce the behavior:

Run a test by taking a screenshot of a large web page (e.g. 1920 × 2483). -> The images match.
Make a change to the front end of the website. .> The image should not match,
Undo the change and make a smaller change until you get a percentage less than 0.01%. -> The change is not detected even though it exists.
Set rawMisMatchPercentage to true. And run the test again. It should now fail with a percentage less than 0.01%.

Expected behavior
There are two ways to handle this if rawMisMatchPercentage is set to false:

One option is to have the test fail and indicate that the failure rate is below 0.00%. I think this would be the correct one.

Another option would be that the test passes, but a warning is thrown in the logs indicating that there is a difference below 0.00%. This should also be indicated in the documentation, so that people do not encounter this confusing behaviour.

Log

With rawMisMatchPercentage set to false:

[2022-04-22T08:47:29.888Z] [0-2]   Then he should see the expected full web-page
[2022-04-22T08:47:29.888Z] [0-2] ✓ Execution successful (28s 487ms)

With rawMisMatchPercentage set to true:

...
[2022-04-26T09:25:03.374Z] [0-0]     ✗ Error: The image tagged as "web-page" does not match its baseline 📸 Mismatch: 0.00014467592592592592%
[2022-04-26T09:25:03.374Z] [0-0] ✗ Execution failed with error (32s 812ms)
...

Multiple element selector only hiding first match

Originally logged here webdriverio/visual-testing#35

Describe your question with as much detail as possible

Hiding only one of the elements matching a selector when updating to puppeteer. It seems the removal/hidden elements is passed through all the way to the client script but I have been having trouble debugging where exactly it might break. My question is basically do I need to debug a visual Chrome execution to get access to the client script logic?

My selector is something like browser.$$('[data-thing="MULTIPLE"]') and I make sure they are all displayable elements before sending them in. When in puppeteer we are getting HTMLElements or are they still WebElements?

If it's about a specific piece of code, try and include some of it to support your question.

https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/clientSideScripts/hideRemoveElements.ts#L30

Environment (please complete the following information):

Node.js version: 10.16.3
NPM version: locally: 6.14.4, docker: 6.9.0
Browser name and version: locally: Chrome 81.0.4044.113, docker: Chrome 79
Platform name and version: locally: OSX, docker: linux
WebdriverIO version: 6.1.0
wdio-image-comparison-service version: 1.13.0

Additional context
Add any other context about the problem here.

CSS breakpoint gets triggert when taking a screenshot

Environment (please complete the following information):

Node.js version: [v13.10.1]
NPM version: [6.14.2]
Browser name and version: [Chrome 80]
Platform name and version: [Windows 10]
Framework name and version: [WDIO V5]
webdriver-image-comparison version: [1.12.2]

Config of the automation framework + plugin
Run programmatically.

Describe the bug
When taking a screenshot, the screen gets resized slightly. If the initial window size is near a css breakpoint, the screenshot might look completly different than expected. (video: https://jmp.sh/JxnTtkA )

To Reproduce
Run the code given below.

Expected behavior
That the screenshot has the initial look

Additional context
Add any other context about the problem here.

const { remote } = require("webdriverio");
const WdioImageComparisonService = require('wdio-image-comparison-service').default;

let env = "asd"
main("Test","ASD")
let testcase = "mainPage"
async function main(testcase, nsc) {
    const browser = await remote({
        "runner": "local",
        "logLevel": "debug",
        "maxInstances": 10,
        "capabilities": {
            "browserName": "chrome",
            'goog:chromeOptions': {
                args: ['--window-size=1020,900'],
            }
        },
        "waitforTimeout": 10000,
        "connectionRetryTimeout": 90000,
        "connectionRetryCount": 3
    });
    let wdioImageComparisonService = new WdioImageComparisonService({
        "savePerInstance": true,
        "clearRuntimeFolder": true,
        "autoSaveBaseline": true,
        "blockOutStatusBar": true,
        "blockOutToolBar": true,
        "disableCSSAnimation": false,
        "fullPageScrollTimeout": 2500,
        "clearFolder": true,
        "formatImageName": `${env.toUpperCase()}_${nsc}_{browserName}_{width}x{height}_${testcase}_{tag}`,
        "screenshotPath": `.tmp/${env.toUpperCase()}/${nsc}`,
    });
    global.browser = browser;
    browser.defaultOptions = wdioImageComparisonService.defaultOptions;
    browser.folders = wdioImageComparisonService.folders
    wdioImageComparisonService.before(browser.capabilities)

    await browser.url("https://saucelabs.com/")

    //Testpart
    let check = await browser.checkScreen("Footer", {
        "fullPageScrollTimeout": 2500
    })
    await browser.deleteSession()
}

ansi-regex vulnerability report

Describe your question with as much detail as possible
A clear and concise question that doesn't require too much conversation. Need more help? Find me on Gitter

Not sure if this is something we can handle on this package but we're getting this vulnerability alert:

Any chance we can handle it on webdriver-image-comparison and wdio-image-comparison-service?

Environment (please complete the following information):

Node.js version: v14.16.1
NPM version: 6.14.12
webdriver-image-comparison version: 0.19.0

Additional context
Add any other context about the problem here.

Error when trying to draw boxes for ignored elements

Environment (please complete the following information):

Node.js version: 13.2.0
NPM version: 6.13.1
Browser name and version: Chrome 74
Platform name and version: Android 9, 10
Framework name and version: WebdriverIO 5.16
Plugin name and version: wdio-image-comparison-service 1.9.0
webdriver-image-comparison version: 0.12.0

Config of the automation framework + plugin
wdio.config.js:

    runner: 'local',
    port: 4723,
    services: [
        [TimelineService],
        [
            'image-comparison', 
            imageCompareService
        ], 
        'appium',
    ],
    specs: [
        join( process.cwd(), './e2e/specs/regression/app/app.*.spec.js')
    ],
    appium: {
        command: "appium",
        logPath : "./logs/appium/android",
    },
    maxInstances: 2,
    specFileRetries: 1, //debug ? 0 : 2,
    capabilities: capabilities,
    logLevel: 'info',
    outputDir: './logs/regression/android',
    bail: 1,
    execArgv: debug ? ['--inspect'] : [],
    waitforTimeout: 20000,
    connectionRetryTimeout: 180000,
    connectionRetryCount: 2,
    framework: 'jasmine',
    reporters: [
        // 'dot',
        'spec',
        ['timeline', { 
            outputDir: join( process.cwd(), './e2e/logs/test-report/android-regression'),
            fileName: 'regression-reporter.html',
            embedImages: true,
            images: {
                quality: 80,
                resize: false,
                reductionRatio: 2
            },
            screenshotStrategy: 'on:error' 
        }],
    ],
    jasmineNodeOpts: {
        defaultTimeoutInterval: debug ? (24 * 60 * 60 * 1000) : 60000,
        expectationResultHandler: function(passed, assertion) {
        }
    }

capabilities:

  {
        maxInstances: 1,
        platformName: "Android",
        platformVersion: "9.0",
        deviceName: "Pixel_2_API_29",
        app: apkPath,
        appPackage: packageName,
        appActivity: "MainActivity",
        automationName: "UiAutomator2",
        nativeWebScreenshot: true,
        avd: "Pixel_2_API_29",
        chromedriverExecutable: join(process.cwd(), "./e2e/common/driver/chromedriver-mac-74"),
        chromedriverPort: 8090,
        systemPort: 8200,
        udid: "Pixel_2_API_29",
        autoGrantPermissions: true,
        avdArgs: "-no-snapshot-save",
        fullReset: true,
    }

image comparison config options:

    baselineFolder: join(screenshotPath, '/baseLine'),
    formatImageName: '{tag}-{width}x{height}',
    screenshotPath: screenshotPath,
    blockOutToolBar: true,
    savePerInstance: true,
    autoSaveBaseline: true,
    blockOutStatusBar: true,
    clearRuntimeFolder: true,
    disableCSSAnimation: true,

Describe the bug
When running test with screenshot comparison on Android (on IOS and browser works fine), I ran into a problem: if screenshots hav some difference I'm getting an error Error: invalid value (typically too big) for the size of the input (surface, pattern, etc.) without any stacktrace or anything, and as a result I'm not getting any diffs or actual screenshots at the end. I've managed to track it down to getAndroidStatusAddressToolBarHeight method: when it computes the statusAddressBarHeight it adds versionOffsets.ADDRESS_BAR when in theory I't shouldn't since I don't have an address bar in my app. And with this value it fails here since ignoredBoxHeight - y becomes negative. When I change negative value for 0 it works as expected.
I've tried to run it on different machines (macos catalina, macos high sierra, win 10) and different android emulators with different chromedriver versions -- doesn't make any difference.

To Reproduce
Steps to reproduce the behavior:

Unfortunately I don't have any exact way to reproduce it. Maybe to initiate new Ionic project with same versions of dependencies:

@ionic/app-scripts 3.2.4
cordova 8.1.2
cordova-android 7.1.1
@angular/core 4.4.6

Expected behavior
When there is difference between baseline and actual screenshots I should get a percentage of difference, actual and diff screenshots saved in specified folders and souldn't get an Error.

Log
wdio logs
appium logs

here

and also some relevant things I've logged while debugging
instanceData: {"browserName":"not-known","browserVersion":"not-known","deviceName":"pixel_2_api_29","logName":"","name":"","nativeWebScreenshot":true,"platformName":"android","platformVersion":"9.0"}

here

OPTIONS: {"isMobile":true,"isViewPortScreenshot":true,"platformName":"android","isAndroidNativeWebScreenshot":true,"blockOutStatusBar":true,"blockOutToolBar":true}

here

navigator.appVersion: 5.0 (Linux; Android 10; Android SDK built for x86 Build/QSR1.190920.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.185 Mobile Safari/537.36, match: Android 10; ,10,,, majorVersion: 10

Additional context
All additional dependencies as they shown in package.json:

"dependencies": {
    "@angular/animations": "^4.4.7",
    "@angular/common": "^4.4.6",
    "@angular/compiler": "^4.4.6",
    "@angular/compiler-cli": "^4.4.6",
    "@angular/core": "^4.4.6",
    "@angular/forms": "^4.4.6",
    "@angular/http": "^4.4.6",
    "@angular/platform-browser": "^4.4.6",
    "@angular/platform-browser-dynamic": "^4.4.6",
    "@angular/tsc-wrapped": "^4.4.6",
    "@ionic-native/android-permissions": "4.3.2",
    "@ionic-native/app-version": "4.3.2",
    "@ionic-native/apple-pay": "4.5.2",
    "@ionic-native/call-number": "4.3.2",
    "@ionic-native/card-io": "4.3.2",
    "@ionic-native/clipboard": "4.3.2",
    "@ionic-native/core": "4.3.2",
    "@ionic-native/device": "4.3.2",
    "@ionic-native/fcm": "4.3.2",
    "@ionic-native/firebase-analytics": "3.14.0",
    "@ionic-native/google-analytics": "4.3.2",
    "@ionic-native/in-app-browser": "4.3.2",
    "@ionic-native/market": "4.5.2",
    "@ionic-native/network": "4.3.2",
    "@ionic-native/push": "4.5.2",
    "@ionic-native/social-sharing": "3.14.0",
    "@ionic-native/splash-screen": "4.3.2",
    "@ionic-native/status-bar": "4.3.2",
    "@ionic-native/unique-device-id": "4.3.2",
    "@ionic/storage": "2.1.3",
    "angular2-moment": "^1.9.0",
    "angular2-text-mask": "^8.0.4",
    "autoprefixer": "^7.2.6",
    "call-number": "^1.0.1",
    "card.io.cordova.mobilesdk": "^2.1.0",
    "cordova-android": "7.1.1",
    "cordova-browser": "^5.0.4",
    "cordova-clipboard": "^1.3.0",
    "cordova-fabric-plugin": "^1.1.14-dev",
    "cordova-ios": "4.5.5",
    "cordova-plugin-android-permissions": "^1.0.0",
    "cordova-plugin-app-version": "^0.1.9",
    "cordova-plugin-applepay": "^3.0.0",
    "cordova-plugin-certificates": "^0.6.4",
    "cordova-plugin-device": "^1.1.7",
    "cordova-plugin-exclude-files": "^0.2.4",
    "cordova-plugin-fcm-with-dependecy-updated": "^2.4.0",
    "cordova-plugin-firebase-analytics": "^0.13.0",
    "cordova-plugin-google-analytics": "^1.8.6",
    "cordova-plugin-inappbrowser": "3.1.0",
    "cordova-plugin-ionic-webview": "^1.1.16",
    "cordova-plugin-localization-strings": "3.2.0",
    "cordova-plugin-market": "^1.2.0",
    "cordova-plugin-network-information": "^1.3.4",
    "cordova-plugin-splashscreen": "^4.1.0",
    "cordova-plugin-statusbar": "^2.4.3",
    "cordova-plugin-whitelist": "^1.3.4",
    "cordova-plugin-x-socialsharing": "^5.4.7",
    "cordova-sqlite-storage": "^2.6.0",
    "cordova-support-google-services": "^1.1.0",
    "cors": "^2.8.4",
    "es6-promise-plugin": "^4.2.2",
    "ionic-angular": "^3.7.1",
    "ionic-img-viewer": "2.8.0",
    "ionic-native": "^2.9.0",
    "ionic-plugin-keyboard": "^2.2.1",
    "ionicons": "3.0.0",
    "jasmine-data-provider": "^2.2.0",
    "mx.ferreyra.callnumber": "0.0.2",
    "phonegap-plugin-multidex": "^1.0.0",
    "phonegap-plugin-push": "2.2.3",
    "prod": "1.0.1",
    "rxjs": "^5.5.8",
    "sw-toolbox": "3.6.0",
    "zone.js": "^0.8.21"
  },
  "devDependencies": {
    "@ionic/app-scripts": "^3.1.8",
    "@types/jasmine": "2.5.41",
    "@types/node": "^10.3.1",
    "angular2-template-loader": "^0.6.2",
    "copy-webpack-plugin": "^4.6.0",
    "html-loader": "^0.5.5",
    "jasmine": "^3.1.0",
    "karma": "^3.0.0",
    "karma-chrome-launcher": "^2.2.0",
    "karma-jasmine": "^1.1.2",
    "karma-jasmine-html-reporter": "^1.1.0",
    "karma-opera-launcher": "^1.0.0",
    "karma-sourcemap-loader": "^0.3.7",
    "karma-webpack": "^3.0.0",
    "null-loader": "^0.1.1",
    "ts-loader": "^3.5.0",
    "typescript": "2.3.4",
    "webpack-merge": "^4.1.5",
    "zone.js": "^0.8.21"
  },

formatImageName docs copy pasta

Pre-check

I'm aware that I can edit the docs and submit a pull request

Describe the improvement

I'd like to report

Unclear documentation
A typo
Missing documentation
Other

Description of the improvement / report
The formatImageName docs appears to be copy pasta as most of the properties are all described as orientation. Also, the defaults for me appear to be empty strings instead of those listed.

checkElement only saves a screenshot with part of the element

Environment (please complete the following information):

Node.js version: 8.11.4 (but also tries in 10.15.0)
Browser name and version: Firefox 67.0 and Chrome 75.0.3770.80 and previous versions
Platform name and version: Debian Jessie (but it's not OS dependent)
Framework name and version: wdio 5.10.0
Plugin name and version: wdio-image-comparison-service 1.1.0
webdriver-image-comparison version: 0.5.0

Config of the automation framework + plugin
See here: https://github.com/anto-ac/checkElement-wdio-image-comparison-service-issue

Describe the bug

When taking a screenshot of an element via checkElement, only part of element is shown in the screenshot.
I think (but I'm not sure) it may happen for elements that are larger than the visible viewport.

See image taken from the library:
https://user-images.githubusercontent.com/30859172/59210996-b7715280-8ba6-11e9-9006-ca2adfa34196.png

To Reproduce
Steps to reproduce the behavior:

Run the test included in this repo:
https://github.com/anto-ac/checkElement-wdio-image-comparison-service-issue

You will need docker and docker-compose to run it.

Expected behavior
The whole element should be captured in the screenshot.

Log
See here: https://gist.github.com/anto-ac/1877d24fa9e88b5fd2ec5acc7b371626

Additional context

Add any other context about the problem here.

clearRuntimeFolder clears for every driver

Environment (please complete the following information):
doesn't matter

Config of the automation framework + plugin

config.services = [
    [
        WdioImageComparisonService.default,
        {
            baselineFolder: join(process.cwd(), './localBaseline/'),
            debug: true,
            formatImageName: '{tag}-{logName}-{width}x{height}',
            screenshotPath: join(process.cwd(), '.tmp/'),
            autoSaveBaseline: true,
            savePerInstance: true,
            blockOutStatusBar: true,
            blockOutToolBar: true,
            clearRuntimeFolder: true,
        },
    ],
]

Describe the bug
When clearRuntimeFolder: true is clears for every instance meaning all folders will be removed during initialisation for each instance. This will lead to folders not holds images of all instances

To Reproduce
See logs below

Expected behavior
Only clear once. This might mean that the cleaning feature needs to be removed from this module and be added to wdio-image-comparison-service because this module doesn't have access to the hooks for WDIO

Log

yarn test.local.desktop               
yarn run v1.22.17
warning ../../../../package.json: No license field
$ wdio tests/configs/wdio.local.desktop.conf.js

Execution of 4 workers started at 2022-07-24T08:54:40.512Z




##############################
!!CLEARING!!
##############################



Failed to connect to selenium. Attempts left: 25 
 connect ECONNREFUSED 127.0.0.1:4444
[0-3] 
[0-3] 
[0-3] 
[0-3] ##############################
[0-3] !!CLEARING!!
[0-3] ##############################
[0-3] 
[0-3] 
[0-3] 
[0-2] 
[0-2] 
[0-2] 
[0-2] ##############################
[0-2] !!CLEARING!!
[0-2] ##############################
[0-2] 
[0-2] 
[0-2] 
[0-1] 
[0-1] 
[0-1] 
[0-1] ##############################
[0-1] !!CLEARING!!
[0-1] ##############################
[0-1] 
[0-1] 
[0-1] 
[0-0] 
[0-0] 
[0-0] 
[0-0] ##############################
[0-0] !!CLEARING!!
[0-0] ##############################
[0-0] 
[0-0] 
[0-0] 
[0-3] RUNNING in chrome - /tests/specs/saveMethodsFolders.spec.js
[0-1] RUNNING in chrome - /tests/specs/desktop.spec.js
[0-2] RUNNING in chrome - /tests/specs/checkMethodsFolders.spec.js
[0-0] RUNNING in chrome - /tests/specs/basics.spec.js
[0-2] PASSED in chrome - /tests/specs/checkMethodsFolders.spec.js
[0-0] PASSED in chrome - /tests/specs/basics.spec.js
[0-1] PASSED in chrome - /tests/specs/desktop.spec.js
[0-3] PASSED in chrome - /tests/specs/saveMethodsFolders.spec.js

 "spec" Reporter:
------------------------------------------------------------------
[chrome 103.0.5060.134 mac os x #0-2] Running: chrome (v103.0.5060.134) on mac os x
[chrome 103.0.5060.134 mac os x #0-2] Session ID: 3555c14211b11b6f3c19e04e160d75e6
[chrome 103.0.5060.134 mac os x #0-2]
[chrome 103.0.5060.134 mac os x #0-2] » /tests/specs/checkMethodsFolders.spec.js
[chrome 103.0.5060.134 mac os x #0-2] wdio-image-comparison-service check methods folder options
[chrome 103.0.5060.134 mac os x #0-2]     checkFullPageScreen method with folder options
[chrome 103.0.5060.134 mac os x #0-2]        ✓ should set all folders using method options
[chrome 103.0.5060.134 mac os x #0-2]
[chrome 103.0.5060.134 mac os x #0-2]     checkScreen method with folder options
[chrome 103.0.5060.134 mac os x #0-2]        ✓ should set all folders using checkScreen method options
[chrome 103.0.5060.134 mac os x #0-2]
[chrome 103.0.5060.134 mac os x #0-2]     checkElement method with folder options
[chrome 103.0.5060.134 mac os x #0-2]        ✓ should set all folders using checkElement method options
[chrome 103.0.5060.134 mac os x #0-2]
[chrome 103.0.5060.134 mac os x #0-2] 3 passing (14.3s)
------------------------------------------------------------------
[chrome 103.0.5060.134 mac os x #0-0] Running: chrome (v103.0.5060.134) on mac os x
[chrome 103.0.5060.134 mac os x #0-0] Session ID: f0cb4ae0c241cbf2707b04caa3956439
[chrome 103.0.5060.134 mac os x #0-0]
[chrome 103.0.5060.134 mac os x #0-0] » /tests/specs/basics.spec.js
[chrome 103.0.5060.134 mac os x #0-0] wdio-image-comparison-service basics
[chrome 103.0.5060.134 mac os x #0-0]     save methods
[chrome 103.0.5060.134 mac os x #0-0]        ✓ should do a save screen
[chrome 103.0.5060.134 mac os x #0-0]        ✓ should do a save element
[chrome 103.0.5060.134 mac os x #0-0]        ✓ should save a fullpage screenshot
[chrome 103.0.5060.134 mac os x #0-0]
[chrome 103.0.5060.134 mac os x #0-0]     check methods
[chrome 103.0.5060.134 mac os x #0-0]        ✓ should fail comparing with a baseline
[chrome 103.0.5060.134 mac os x #0-0]
[chrome 103.0.5060.134 mac os x #0-0] 4 passing (12.8s)
------------------------------------------------------------------
[chrome 103.0.5060.134 mac os x #0-1] Running: chrome (v103.0.5060.134) on mac os x
[chrome 103.0.5060.134 mac os x #0-1] Session ID: c1c95a4e424646897dcb2c06f1648c4b
[chrome 103.0.5060.134 mac os x #0-1]
[chrome 103.0.5060.134 mac os x #0-1] » /tests/specs/desktop.spec.js
[chrome 103.0.5060.134 mac os x #0-1] wdio-image-comparison-service desktop
[chrome 103.0.5060.134 mac os x #0-1]     compare element
[chrome 103.0.5060.134 mac os x #0-1]        ✓ should compare successful with a baseline
[chrome 103.0.5060.134 mac os x #0-1]
[chrome 103.0.5060.134 mac os x #0-1]     compare full page
[chrome 103.0.5060.134 mac os x #0-1]        ✓ should compare successful with a full page desktop baseline
[chrome 103.0.5060.134 mac os x #0-1]
[chrome 103.0.5060.134 mac os x #0-1]     compare tabbable
[chrome 103.0.5060.134 mac os x #0-1]        ✓ should compare successful with a tabbable desktop baseline
[chrome 103.0.5060.134 mac os x #0-1]
[chrome 103.0.5060.134 mac os x #0-1] 3 passing (19s)
------------------------------------------------------------------
[chrome 103.0.5060.134 mac os x #0-3] Running: chrome (v103.0.5060.134) on mac os x
[chrome 103.0.5060.134 mac os x #0-3] Session ID: 3ae4f4e26035640d7503f0616cb649cc
[chrome 103.0.5060.134 mac os x #0-3]
[chrome 103.0.5060.134 mac os x #0-3] » /tests/specs/saveMethodsFolders.spec.js
[chrome 103.0.5060.134 mac os x #0-3] wdio-image-comparison-service save methods folder options
[chrome 103.0.5060.134 mac os x #0-3]     saveFullPageScreen method with folder options
[chrome 103.0.5060.134 mac os x #0-3]        ✓ should set folders using method options
[chrome 103.0.5060.134 mac os x #0-3]        ✓ should set folders using default options
[chrome 103.0.5060.134 mac os x #0-3]
[chrome 103.0.5060.134 mac os x #0-3]     saveScreen method with folder options
[chrome 103.0.5060.134 mac os x #0-3]        ✓ should set folders using saveScreen method options
[chrome 103.0.5060.134 mac os x #0-3]        ✓ should set folders using saveScreen default options
[chrome 103.0.5060.134 mac os x #0-3]
[chrome 103.0.5060.134 mac os x #0-3]     saveElement method with folder options
[chrome 103.0.5060.134 mac os x #0-3]        ✓ should set folders using saveElement method options
[chrome 103.0.5060.134 mac os x #0-3]        ✓ should set folders using saveElement default options
[chrome 103.0.5060.134 mac os x #0-3]
[chrome 103.0.5060.134 mac os x #0-3] 6 passing (22s)


Spec Files:      4 passed, 4 total (100% completed) in 00:00:35 

✨  Done in 41.23s.

Throw proper error when element height/width is 0

Environment (please complete the following information):
for all

Config of the automation framework + plugin
not needed

Describe the bug
Fix webdriverio/visual-testing#90

To Reproduce
See linked issue

Expected behavior
Throw an error with a proper message about 0 height/width

Browser platform and version variables for formatImageName

Hi again; I'd like to save the browser platform and major version as part of the screenshot filename. Variables for these values which could be used in formatImageName would be perfect, but I'm open to other solutions as well. Thanks for your time!

Use case for hideElements

I have a page that contains the following snippet:

<div id="infoGraphics">....</div>

Using chrome devtools:

document.querySelectorAll('div#infoGraphics').length
-> 1

In writing my test I have a getter in my page object like:

get infoGraphic() { return $('div#infoGraphics'); }

And my test:

browser.saveScreen('LoginPage', { hideElements: [ LoginPage.infoGraphic ] });

Which fails with:

Cannot set property 'visibility' of undefined

However, if I use the following, my test will work:

get infoGraphic() { return $$('div#infoGraphics'); }

Is this expected? Or have I misused the API?

checkFullPageScreen method is not calculate correctly the scroll height

Environment:

Node.js version: 8.12.0
NPM version: 6.4.1
Browser name and version: Chrome 72.0.3626.96
Platform name and version: Ubuntu 16.04
Framework name and version: Protractor 5.4.2
Plugin name and version: protractor-image-comparison 3.2.0
webdriver-image-comparison version: 0.5.0

Config of the automation framework + plugin
here a basic example of how my conf.js looks like:

exports.config = {
   framework: 'jasmine',
   multiCapabilities: env.multiCapabilities,
   baseUrl: env.baseUrl,
   specs: [
     'image-comparator.spec.js',
   ],
   SELENIUM_PROMISE_MANAGER: false,
   directConnect: true,
}

Describe the bug
Using the checkFullPageScreen method with a dialog which blocks the browser's scroll, I am getting a weird behavior as the way of calculating the scroll height is not working well.

To Reproduce
I created this simple example: https://angular-5sbdat.stackblitz.io/

and this is basic my test:

describe('compare example screen', () => {
  beforeEach(async () => {
    await browser.waitForAngularEnabled(false);
    await browser.get("https://angular-5sbdat.stackblitz.io/");
  });

  it('should compare login page with a baseline', async () => {
    await browser.sleep(3000);
    await element.all(by.css('button')).get(0).click();
    expect(await browser.imageComparison.checkFullPageScreen('test')).toEqual(0);
  });
});

Using the version 0.4.8 I am getting this result (the desired one):

However using the version 0.5.0:

As you can see, using the version 0.5.0 checkFullPageScreen method is scrolling down but it does not make sense because the scroll is blocked by the dialog.

How to prevent memory leaks when initializing mutliple browsers

Describe your question with as much detail as possible
When having a loop which creates browsers and takes screenshots, after the session/iteration was closed, a chunk of memory is still being used and not garbage collected.

After the first iteration:

After 6 iterations

Is there a way to clean up this memory? Thanks and have a great day!

If it's about a specific piece of code, try and include some of it to support your question.
Code used to create the Heapsnapshots:

const { remote } = require("webdriverio");
const WdioImageComparisonService = require('wdio-image-comparison-service').default;
let folderActual
let browsers = []
const heap = require("heapdump")
let testBrowser
let testResults
let env
let window
callTests(new Array(10)).then(erg => {
    res(erg)
})
async function main(nsc, i) {
    //console.log(`Started test: ${testcase}@${nsc} (${i+1}/${allNSC.length})`)
    const browser = await remote({
        capabilities : {
            browserName: "chrome",
        },
        logLevel: "error",
        outputDir: __dirname + "../../../logs/"
    }).catch(err => console.log(err));
    browsers[i] = browser
    let wdioImageComparisonService = new WdioImageComparisonService({
        "savePerInstance": true,
        "clearRuntimeFolder": true,
        "autoSaveBaseline": true,
        "blockOutStatusBar": true,
        "blockOutToolBar": true,
        "disableCSSAnimation": false,
        "fullPageScrollTimeout": 2500,
        "clearFolder": true,
        formatImageName: `{tag}`,
        screenshotPath: `.tmp/`,
    });

    global.browser = browser;
    browser.defaultOptions = wdioImageComparisonService.defaultOptions;
    browser.folders = wdioImageComparisonService.folders
    wdioImageComparisonService.before(browser.capabilities)

    let url = "https://www.nytimes.com/"
    await browser.url(url)
    await heap.writeSnapshot(__dirname + "/../heaps/"+i+"before.heapsnapshot")
    let check = await browser.checkFullPageScreen(i, {
        "fullPageScrollTimeout": 2500
    })
    await heap.writeSnapshot(__dirname + "/../heaps/"+i+"after.heapsnapshot")

    await browser.deleteSession()
    await delete browser
    await delete browsers[i]
}

function callTests(arr, i) {
    return new Promise(res => {
        if (i == undefined) i = 0
        let error = false
        main(arr[i], i).catch(async e => {
            console.error(e, arr[i])
            error = e
            await browsers[i].deleteSession();
            await delete browsers[i]
        }).then(async erg => {
            if (i < arr.length - 1) return res(callTests(arr, ++i))
            else res()
        });
    })
}

function setupTest(testdata) {
    env = testdata.env
    allNSC = testdata.nscs
    testBrowser = testdata.browser
    testrunId = testdata.testrunId
    port = testdata.port
    testrunName = testdata.name
    window = testdata.window
    config.remote.capabilities = {
        browserName: testBrowser,
        //'goog:chromeOptions': {
        //    args: ['--window-size=' + testdata.window.width + ',' + testdata.window.height],
        //}
    }
    testResults = { testcase: testcase, teststeps: 1, teststepNames: ["mainPage"], data: [], browser: testBrowser }
    return new Promise((res, rej) => {
        
    })
}
module.exports.setupTest = setupTest

Environment (please complete the following information):

Node.js version: [13.10.1]
NPM version: [6.14.5]
Browser name and version: [Chrome 83]
Platform name and version: [Windows 10]
webdriver-image-comparison version: [1.12.2]

Option to keep checking image differences until it stabilizes

Is your feature request related to a problem? Please describe.
I'm working on a web page with big slow images. The comparison tool sometimes returns me false negatives because the image has not been completely loaded whenever the comparison is made.

Describe the solution you'd like
I can think of two possible solutions:

Webdriver-image-comparison waits for the image to be completely loaded (don't know if that's even possible)
Less elegant, is to wait some time after a negative comparison and then compare the image again. If the result of both comparisons is the same, we could assume that the image has not changed and hence fail the comparison. If the result is different, then the first image and the second one are different and we can assume the image is still loading. In that case, I would continue with the algorithm until the image matches or stabilizes.

Note this second option can be implemented from the npm package consumer, I did a very quick and dirty approach to see if it worked (and it works on my machine). The problems with my bold implenetation are:

Nobody else can benefit from the feature
It leaves images for intermediate failed comparisons in the output folder that should not be left there in an ideal solution

The function, in case it helps:

async StabilizeAndCompareImage(attempt = 0)
    {
        console.log("Attempt "+ attempt);
        
        var initialComparisonResult = await browser.imageComparison.checkScreen('login', { autoSaveBaseline: true });
        if(initialComparisonResult === 0)
        {
            console.log("Attempt "+ attempt + ". Image as expected.");
            return initialComparisonResult;
        }
        else{
            await browser.sleep(100);
            var newComparisonResult = await browser.imageComparison.checkScreen('login', { autoSaveBaseline: true });
            if(newComparisonResult == initialComparisonResult)
            {
                console.log("Attempt "+ attempt + ". Image not as expected, but has not changed.");
                return newComparisonResult;
            }
            else{
                console.log("Attempt "+ attempt + ". Image not as expected and it's changing, so trying again.");
                return this.StabilizeAndCompareImage(attempt + 1);
            }
        }
    }

Describe alternatives you've considered
I found this problem with protractor-image-comparison package. First headed there to see if the solution is feasible and then decided that it's a best fit for webdriver-image-comparison.

Additional context
Nothing to add.

hideAfterFirstScroll giving error in typescript Spec

Copy of webdriverio/visual-testing#43

======================================================================
Trying to use hideAfterFirstScroll in checkFullPageScreen method but it's giving error:
TS2740: Type 'Element' is missing the following properties from type 'HTMLElement': accessKey, accessKeyLabel, autocapitalize, dir, and 230 more.

expect(browser.checkFullPageScreen(this.test.title,{hideAfterFirstScroll: [$('#privacy-information')]} )).equal(0);

TypeScript
Node.js version: [v12.13.1]
NPM version: [6.12.1]
Browser name and version: [NA] - compiler issue
Platform name and version: [NA] - compiler issue
WebdriverIO version: [5.14.0]
wdio-image-comparison-service version: [0.14.4]
======================================================================

Support big screenshot

Environment (please complete the following information):

webdriver-image-comparison version: 0.13.2

Describe the bug
Web driver Image fail on big images.

Given error :

invalid value (typically too big) for the size of the input (surface, pattern, etc.)

Automattic/node-canvas#1374

To Reproduce
Try with an image ~40000px long.

Expected behavior
webdriver-image-comparison should be able to manage big images (useful for full page screen shot).

Log

invalid value (typically too big) for the size of the input (surface, pattern, etc.)

Additional context
this comment advice to split the image to avoid this issue.

Disabling CSS animation doesn't block animations for pseudo elements

Environment (please complete the following information):

webdriver-image-comparison version: 2.0.0

Describe the bug
Hi there! Thanks a lot for the package!
While using it for visual regression tests in our UI components library we've discovered that we have tests breaking for animated components in certain cases even though disableCSSAnimation option is set for true.
With a bit of digging trying to debug the problem, we've discovered that custom styles injected to disable CSS animation don't block animations for pseudo elements. And this is where we sometimes have loading spinners added to components and other animated stuff.
I think having animated pseudo elements is pretty common in frontend development so I would expect the package to disable animations for them as well.

Expected behavior
In here:

webdriver-image-comparison/lib/clientSideScripts/setCustomCss.ts

Line 11 in 980a7d4

const disableTransformationsTransitionsAnimations = `

I would propose adding *::before, *::after to existing * selector so animation and transition blocking also affects pseudo elements.

Happy to create a pull request if you think this is the way to go, otherwise we would go custom and inject our own custom styles for those test cases.

blockOut option is not working correctly

Environment:

Node.js version: 8.12.0
NPM version: 6.4.1
Browser name and version: Chrome 72.0.3626.96
Platform name and version: Ubuntu 16.04
Framework name and version: Protractor 5.4.2
Plugin name and version: protractor-image-comparison 3.0.1
webdriver-image-comparison version:0.3.0

Config of the automation framework + plugin
here a basic example of how my conf.js looks like:

exports.config = {
   framework: 'jasmine',
   multiCapabilities: env.multiCapabilities,
   baseUrl: env.baseUrl,
   specs: [
     'image-comparator.spec.js',
   ],
   SELENIUM_PROMISE_MANAGER: false,
   directConnect: true,
}

Describe the bug
I'm trying to use the method checkFullPageScreen with the option blockOut, bu it turns out that the comparator is not able to detect any change. In theory, there are many differences between the baseline and the actual version and I wanted to exclude one of them using the blockOut option.

However if I use the method checkFullPageScreen without the option blockOut, the comparator detects all the existed changes.

To Reproduce
I used this code:

expect(await browser.imageComparison.checkFullPageScreen('profile', {
            blockOut: [{
              height: 48,
              width: 48,
              x: 1817,
              y: 16
            }],
})).toEqual(0);

it is not able to detect any change, even worse, this code neither:

expect(await browser.imageComparison.checkFullPageScreen('profile', {
            blockOut: [{
              height: 0,
              width: 0,
              x: 0,
              y: 0
            }],
})).toEqual(0);

However this one, yes:

expect(await browser.imageComparison.checkFullPageScreen('profile')).toEqual(0);

Android 10 is not supported

Android 10 is not supported and this makes the module break if checkElement||checkFullPageScreen screenshots are taken. This can be fixed by adding a new property to the https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/helpers/constants.ts#L29 Android offsets

New iPhone 14 and iOS 16 came out

The new iPhone 14/Pro/Pro max, together with iOs 16 came out. We need to add the following dimensions

iPhone 14: 390x844 => same as 12|12Pro|13|13Pro
iPhone 14 Plus: 428x926 => same as 12ProMax|13ProMax
iPhone 14 Pro: 393x852 => NEW
iPhone 14 Pro Max: 430x932 => NEW

to this file https://github.com/wswebcreation/webdriver-image-comparison/blob/main/lib/helpers/constants.ts

and check if everything still works with iOS 16

Feature Request: Screenshot fail gracefully when removeElements non-existing

Is your feature request related to a problem? Please describe.
I have elements that only exist on certain pages and the library not finding one of the remove elements causes the screenshot to never be taken. For me, it isn't critical that every element in removeElements is removed. I just want the elements that do exist to be removed and non-existing ones to be ignored because some can be intermittent (ex: random sales popup).

Describe the solution you'd like
I would like the screenshot function (like saveFullPageScreen()) to first check if an element exists before attempting to remove it, or at least fail in a way that doesn't stop the screenshot from being taken.

Describe alternatives you've considered
My current solution is to create a removeElements array ahead of the saveFullPageScreen() by first checking isDisplayed() on each element before passing that to the saveFullPageScreen() options.

Additional context
It looks like xPath doesn't like elements that don't exist, causes the function to fail. For context in the error below .lp_promo_banner is the element by trying to hide that doesn't exist on all pages.

If it's useful, my (simplified) function call looks like:

browser.saveFullPageScreen(path, {
    removeElements: [
        $('.lp_promo_banner')
    ]
})

Not working with WinAppDriver because browser.execute is not implemented

Environment:

Node.js version: 16.15.0
NPM version: 8.5.5
Browser name and version: ApplicationName
Platform name and version: Windows
Framework name and version: Webdriver.IO 7.16.4
Plugin name and version: [e.g. protractor-image-comparison 3.0.0]
wdio-image-comparison-service: 3.1.1

Config of the automation framework + plugin

services: [ [
     'image-comparison',
     {
       // Some options, see the docs for more
       baselineFolder: path.join(process.cwd(), './datas/RefImages/'),
       formatImageName: '{tag}',
       screenshotPath: path.join(process.cwd(), '.tmp/'),
       savePerInstance: true,
       autoSaveBaseline: true,
       blockOutStatusBar: true,
       blockOutToolBar: true,
       // ... more options
     },
   ],
],
capabilities: [
     // CAPA for local test
     platformName: 'WINDOWS',
     browserName: `DieManufacturing.${dieApplication}`,
     'ms:experimental-webdriver': true,
     'ms:waitForAppLaunch': '10',
     'appium:app': `C:/Program Files/Intescia/DieManufacturing.${dieApplication}/Fr.Intescia.DieManufacturing.Client.exe`,
     'appium:appArguments': '-noCloseConfirmationPopUp',
     'appium:appiumOptions': {
       w3c: false,
       'ms:experimental-webdriver': true,
     },
],

Describe the bug
Unable to use webdriver-image-comparaison with WinAppDriver because browser.execute command is not implemented in this driver.
Could you develop a mode that does not use this method(brower.execute)?

Expected behavior
An option to disable the calls to browser.execute

Log

2022-05-04T09:03:51.386Z INFO webdriver: COMMAND $("~_rowHeaderCheckBoxLine_2")
[0-0] 2022-05-04T09:03:51.386Z INFO webdriver: COMMAND findElement("accessibility id", "_rowHeaderCheckBoxLine_2")
[0-0] 2022-05-04T09:03:51.386Z INFO webdriver: [POST] http://127.0.0.1:4723/session/bc1a8f39-ac62-460d-b860-f58ab5ec9a60/element
[0-0] 2022-05-04T09:03:51.386Z INFO webdriver: DATA { using: 'accessibility id', value: '_rowHeaderCheckBoxLine_2' }
[0-0] 2022-05-04T09:03:52.075Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': '7.25576.48105459',
[0-0]   ELEMENT: '7.25576.48105459'
[0-0] }
[0-0] 2022-05-04T09:03:52.078Z INFO webdriver: RESULT Element {
[0-0]   sessionId: 'bc1a8f39-ac62-460d-b860-f58ab5ec9a60',
[0-0]   elementId: '7.25576.48105459',
[0-0]   ELEMENT: '7.25576.48105459',
[0-0]   selector: '~_rowHeaderCheckBoxLine_2',
[0-0]   parent: Browser {
[0-0]     sessionId: 'bc1a8f39-ac62-460d-b860-f58ab5ec9a60',
[0-0]     capabilities: {
[0-0]       platformName: 'WINDOWS',
[0-0]       browserName: 'DieManufacturing.Demo',
[0-0]       'ms:experimental-webdriver': true,
[0-0]       'ms:waitForAppLaunch': 10,
[0-0]       app: 'C:/Program Files/Intescia/DieManufacturing.Demo/Fr.Intescia.DieManufacturing.Client.exe',
[0-0]       appArguments: '-noCloseConfirmationPopUp',
[0-0]       appiumOptions: [Object]
[0-0]     },
[0-0]     addCommand: [Function (anonymous)],
[0-0]     overwriteCommand: [Function (anonymous)],
[0-0]     addLocatorStrategy: [Function (anonymous)],
[0-0]     config: {
[0-0]       specs: [Array],
[0-0]       suites: [Object],
[0-0]       exclude: [],
[0-0]       outputDir: undefined,
[0-0]       logLevel: 'debug',
[0-0]       logLevels: {},
[0-0]       excludeDriverLogs: [],
[0-0]       bail: 0,
[0-0]       waitforInterval: 800,
[0-0]       waitforTimeout: 30000,
[0-0]       framework: 'cucumber',
[0-0]       reporters: [Array],
[0-0]       services: [Array],
[0-0]       maxInstances: 16,
[0-0]       maxInstancesPerCapability: 12,
[0-0]       filesToWatch: [],
[0-0]       connectionRetryTimeout: 240000,
[0-0]       connectionRetryCount: 0,
[0-0]       execArgv: [],
[0-0]       runnerEnv: {},
[0-0]       runner: 'local',
[0-0]       specFileRetries: 0,
[0-0]       specFileRetriesDelay: 0,
[0-0]       specFileRetriesDeferred: false,
[0-0]       reporterSyncInterval: 100,
[0-0]       reporterSyncTimeout: 5000,
[0-0]       cucumberFeaturesWithLineNumbers: [],
[0-0]       autoCompileOpts: [Object],
[0-0]       mochaOpts: [Object],
[0-0]       jasmineOpts: [Object],
[0-0]       cucumberOpts: [Object],
[0-0]       onPrepare: [],
[0-0]       onWorkerStart: [],
[0-0]       before: [Array],
[0-0]       beforeSession: [],
[0-0]       beforeSuite: [],
[0-0]       beforeHook: [],
[0-0]       beforeTest: [Array],
[0-0]       beforeCommand: [],
[0-0]       afterCommand: [],
[0-0]       afterTest: [],
[0-0]       afterHook: [],
[0-0]       afterSuite: [],
[0-0]       afterSession: [],
[0-0]       after: [],
[0-0]       onComplete: [],
[0-0]       onReload: [],
[0-0]       beforeFeature: [Array],
[0-0]       beforeScenario: [Array],
[0-0]       beforeStep: [],
[0-0]       afterStep: [Array],
[0-0]       afterScenario: [Array],
[0-0]       afterFeature: [Array],
[0-0]       inspect: false,
[0-0]       port: 4723,
[0-0]       coloredLogs: true,
[0-0]       screenshotPath: './errorShots/',
[0-0]       baseUrl: '',
[0-0]       params: [Object],
[0-0]       _: [Array],
[0-0]       spec: [Array],
[0-0]       '$0': 'node_modules\\@wdio\\cli\\bin\\wdio.js',
[0-0]       ignoredWorkerServices: []
[0-0]     },
[0-0]     params: {
[0-0]       scenarioContext: {},
[0-0]       scenarioContext2: {},
[0-0]       featureContext: {},
[0-0]       importContractName: '',
[0-0]       importContractsList: [],
[0-0]       url: '',
[0-0]       profiles: [],
[0-0]       screenshotName: '',
[0-0]       fusion: [],
[0-0]       contactsName: [],
[0-0]       contractInformation: {},
[0-0]       orderedDieList: [],
[0-0]       tableColumnsList: [],
[0-0]       DIEVersion: '0000253.01.1'
[0-0]     }
[0-0]   },
[0-0]   emit: [Function: bound ],
[0-0]   isReactElement: false,
[0-0]   addCommand: [Function (anonymous)],
[0-0]   overwriteCommand: [Function (anonymous)]
[0-0] }
[0-0] 2022-05-04T09:03:52.078Z INFO webdriver: COMMAND checkElement(<object>, "ligne2_selected")
[0-0] 2022-05-04T09:03:52.081Z INFO webdriver: COMMAND executeScript(<fn>, <object>)
[0-0] 2022-05-04T09:03:52.081Z INFO webdriver: [POST] http://127.0.0.1:4723/session/bc1a8f39-ac62-460d-b860-f58ab5ec9a60/execute
[0-0] 2022-05-04T09:03:52.081Z INFO webdriver: DATA {
[0-0]   script: 'return (function hideScrollBars(hide) {\n' +
[0-0]     '    try {\n' +
[0-0]     '        if (hide) {\n' +
[0-0]     "            document.body.style.overflow = 'hidden';\n" +
[0-0]     '        }\n' +
[0-0]     '        else {\n' +
[0-0]     "            document.body.style.overflow = '';\n" +
[0-0]     '        }\n' +
[0-0]     '    }\n' +
[0-0]     '    catch (e) {\n' +
[0-0]     '    }\n' +
[0-0]     '}).apply(null, arguments)',
[0-0]   args: [ true ]
[0-0] }
[0-0] 2022-05-04T09:03:52.096Z ERROR webdriver: Request failed with status 405 due to unknown method: Method is not implemented
[0-0] Error in "0: And I expect that element "~_rowHeaderCheckBoxLine_2" is compared successful with the file "ligne2_selected""
NotImplementedError: Method is not implemented
    at WindowsDriver.execute (C:\2-gitlab\end-to-end-tests-wdio\node_modules\appium-windows-driver\lib\commands\execute.js:26:9)
    at commandExecutor (C:\2-gitlab\end-to-end-tests-wdio\node_modules\appium-base-driver\lib\basedriver\driver.js:335:9)
    at C:\2-gitlab\end-to-end-tests-wdio\node_modules\async-lock\lib\index.js:146:12
    at AsyncLock._promiseTry (C:\2-gitlab\end-to-end-tests-wdio\node_modules\async-lock\lib\index.js:280:31)
    at exec (C:\2-gitlab\end-to-end-tests-wdio\node_modules\async-lock\lib\index.js:145:9)
    at AsyncLock.acquire (C:\2-gitlab\end-to-end-tests-wdio\node_modules\async-lock\lib\index.js:162:3)
    at WindowsDriver.executeCommand (C:\2-gitlab\end-to-end-tests-wdio\node_modules\appium-base-driver\lib\basedriver\driver.js:348:39)
    at AppiumDriver.executeCommand (C:\2-gitlab\end-to-end-tests-wdio\node_modules\appium\lib\appium.js:563:36)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at asyncHandler (C:\2-gitlab\end-to-end-tests-wdio\node_modules\appium-base-driver\lib\protocol\protocol.js:297:21)
[0-0] 2022-05-04T09:03:52.096Z DEBUG webdriver: request failed due to response error: unknown method
[0-0] 2022-05-04T09:03:52.097Z DEBUG @wdio/utils:shim: Finished to run "afterStep" hook in 0ms
[0-0] Save a screenshot
[0-0] 2022-05-04T09:03:52.103Z INFO webdriver: COMMAND takeScreenshot()
[0-0] 2022-05-04T09:03:52.103Z INFO webdriver: [GET] http://127.0.0.1:4723/session/bc1a8f39-ac62-460d-b860-f58ab5ec9a60/screenshot
[0-0] 2022-05-04T09:03:52.339Z INFO webdriver: RESULT iVBORw0KGgoAAAANSUhEUgAADxIAAAhQCAIAAAA7KwjGAAAAAXNSR0IArs4c6...
[0-0] 2022-05-04T09:03:52.343Z DEBUG @wdio/utils:shim: Finished to run "afterScenario" hook in 0ms
[0-0] 2022-05-04T09:03:52.345Z INFO webdriver: COMMAND sendKeys(<object>)
[0-0] 2022-05-04T09:03:52.345Z INFO webdriver: [POST] http://127.0.0.1:4723/session/bc1a8f39-ac62-460d-b860-f58ab5ec9a60/keys
[0-0] 2022-05-04T09:03:52.345Z INFO webdriver: DATA { value: [ '', '1', '' ] }
[0-0] 2022-05-04T09:03:52.655Z DEBUG @wdio/utils:shim: Finished to run "afterFeature" hook in 311ms
[0-0] 2022-05-04T09:03:52.669Z INFO webdriver: COMMAND deleteSession()
[0-0] 2022-05-04T09:03:52.669Z INFO webdriver: [DELETE] http://127.0.0.1:4723/session/bc1a8f39-ac62-460d-b860-f58ab5ec9a602022-05-04T09:03:53.043Z DEBUG @wdio/local-runner: Runner 0-0 finished with exit code 1

Additional context
Add any other context about the problem here.

tslint doesn't work

In order to use tslint, I had to manually copy packages vrsource-tslint-rules and tslint-eslint-rules to /node_modules/webdriver-image-comparison/node_modules.
Otherwise, I got an error:

$ tslint --fix '**/*.ts' && tsc --project tsconfig.json --noEmit
Failed to load .../offer-checkout-2.0/node_modules/webdriver-image-comparison/tslint.json: Could not find custom rule directory: node_modules/vrsource-tslint-rules/rules

Also, I had to manually install vrsource-tslint-rules and tslint-eslint-rules.

Support Elements objects ($$) as members of hideElements and removeElements arrays

It doesn't seem possible to pass Elements objects ($$) as members of the hideElements array. The following error is produced:

javascript error: Cannot set property 'visibility' of undefined

I ended up coming up with a workaround, but it might be more intuitive to allow this.

The ES6 workaround I discovered:

let result = browser.checkElement(timeline.strip, "Timeline strip", {
  hideElements: [].concat(...[
    timeline.controls.panel,
    timeline.ticks, // $$ object
    timeline.labels // $$ object
  ])
});

Thanks for your work on this by the way, it's immediately immensely useful as I can now scrap the v4 fork of my suite I was keeping alive just for visual regression.

wswebcreation / webdriver-image-comparison Goto Github PK

webdriver-image-comparison's Introduction

webdriver-image-comparison

TO-DO

webdriver-image-comparison's People

Contributors

Stargazers

Watchers

Forkers

webdriver-image-comparison's Issues

Recommend Projects

Recommend Topics

Recommend Org