jnormington / dotdiff Goto Github PK

The gem works fine with JRuby, thanks! :)

The first time a test is run, an image is generated (btw, it might be a good idea to document this as I was confused in the beginning as to how to generate the screenshots to compare against). The second time it is run, the following error appears:

expected Capybara::Session to match image but failed with: 0compare: unable to open image `.../failures/some-image.diff.png': No such file or directory @ error/blob.c/OpenBlob/2701.

I can see that in that directory a .new.png file is generated, but no .diff.png and hence the exception. The funny thing is that when a test is run for the third+ time, everything works fine - a .diff.png file is generated and the test passes.

Not sure if this is a problem only on JRuby.

JRuby doesn't support rmagick (read here) but it does offer alternatives: RMagick4J. I guess it shouldn't be too hard to check the runtime and require only the gem for the specific platform. Can you do that?

Main downside of not being able to set percentage instead of pixels for difference threshold is an issue I am having with current Visual Regression (VR) tests I have.

Example:
I am VR testing a button with 3000 pixels total >> test failed because ~600 pixels difference.
I was going to increase the threshold as button is okay to pass with this difference but new threshold will not make sense to baseline images with higher pixels such as one that I have that has 400000 where test fail because of ~52000 difference. Increasing this threshold will make the button VR test pass all the time which won't be correct.

I'm testing all those XPATHs in the browser with $x("some-xpath"). One example is or clauses:

//div[contains(text(), 'a')]/following-sibling::div
//div[contains(text(), 'b')]/following-sibling::div
//div[contains(text(), 'c')]/following-sibling::div

All of these work by themselves (both in the browser and in xpath_elements_to_hide). However if I do it the following way:

//div[contains(text(), 'a') or contains(text(), 'b') or contains(text(), 'c')]/following-sibling::div

Which is equivalent to the above three cases. It works in the browser but doesn't work when I put it in xpath_elements_to_hide, i.e. the elements are still visible in the screenshot.

Another example is this:

//div[contains(@class, 'qwe')]/div[1]

Not sure why it doesn't hide the element.

Any idea what I might be doing wrong or where the bug is?

There was a prod bug where the component's layout/sizing didn't change, but the background color changed to a darker version. I created a VR test to check if this would've fail the test, but it doesn't. Will there be a way to catch this color changes as well and fail the test when they occur.

Example of issue that it's not failing the VR test when comparing these two images:
PROD bug Image: https://www.dropbox.com/s/vx8goc9pi6nxl2e/small_product_square_slice.png?dl=0
Correct Image: https://www.dropbox.com/s/qsr7efo2pea2ioe/small_product_square_slice.new.png?dl=0