protractor-flake does not retry failed specs if the expectation is in a promise about protractor-flake HOT 6 CLOSED

This is because the stack trace for exceptions thrown in webdriver promises does not include the line of the file. Protractor flake takes a very simple approach to scrapping through the stacktraces and if this information isn't found it treats it as a re-run.

There isn't a great solve for this at the moment; I'd more than welcome a PR to make this better.

Two questions

• Can you share the output of the exception without using protractor flake?
• Can you share the options you are passing to protractor-flake?

from protractor-flake.

Output:

   The Management Basic Onboarding should
     ✓ start an onboarding tour
 .    ✓ continue the onboarding process on product page
 F    ✗ should move to the next step when navigating between pages
       - Expected 'basicManagementTour:2:0' to be 'basicManagementTour:1'.
           at /home/jenkins/workspace/TA-3-test-chrome/viewtest/tests/onboarding/ManagementBasicOnboardingSpec.js:37:31
           at ManagedPromise.invokeCallback_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:1379:14)
           at TaskQueue.execute_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2913:14)
           at TaskQueue.executeNext_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2896:21)
           at asyncRun (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2775:27)
           at /usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:639:7
           at process._tickCallback (internal/process/next_tick.js:103:7)

Strange thing is ... I do see a filename to parse here.

Command line options:
protractor-flake --max-attempts=3 -- protractor.conf.js

This is because the stack trace for exceptions thrown in webdriver promises does not include the > line of the file. Protractor flake takes a very simple approach to scrapping through the
stacktraces and if this information isn't found it treats it as a re-run.

Maybe a quick fix would be that all the tests are being retried when the number of tests to retry don't match with the number of failed tests. I don't care if the tests have to take a while because they are all being retried, but failed tests not being retried while in the end the entire test suite is seen as 'green' is a bit dangerous ...

from protractor-flake.

Ah interesting; this output is from a different reporter than I am used to; unfortunately it still doesn't pass the criteria that the standard parser looks for.

Looking for passed files would be great but I don't know of a generic way to get which files passed and which did not (protractor doesn't provide this as far as I know). A custom reporter might solve this. Again, would love any ideas or a PR here.

I am going to work on writing the ability to make custom parsers for cases like this (since it seems like the spec file should always be the 1st at /stacktrace/path in your case. That way you can tailor it to your output and (hopefully) get the desired result. Does that sound acceptable?

from protractor-flake.

Sounds like a good idea.

from protractor-flake.

I have a pull request for custom parsers #45; I'd appreciate it if you'd take a look!

from protractor-flake.

Custom parsers have been released 9791183. I'm going to go ahead and close this for now. Feel free to reopen another issue if there is still a problem.

from protractor-flake.