Assume/Abort in sub Steps creates Passed Scenario in the report with odd count of Steps/Skipped about cluecumber HOT 11 CLOSED

Hi!
Thanks for quick handling!
Added .txt to JSON (It contains an image too, but should be ok)

Console-Out.txt
cucumber.json.txt

from cluecumber.

@bischoffdev the state of a scenario is the most severe status of it's steps. But I didn't look at the json, maybe I'm reading your comment wrong.

https://github.com/cucumber/cucumber-jvm/blob/main/cucumber-core/src/main/java/io/cucumber/core/runner/TestCaseState.java#L54

from cluecumber.

I just tried Allure-reporting and got this for an Assume()/Abort() operation:
(w/o any claims of what is wrong or right, just as info)

from cluecumber.

Thank you for reporting, I will look into this.

from cluecumber.

A scenario is considered skipped when all steps are skipped. When there are passing steps before it is considered passed. This is according to the official Cucumber guidelines.

(see issue #283)

Could you send me the JSON result file of your use case so I can check this better?

from cluecumber.

Your example shows a passed step and then a skipped one. According to the Cucumber specs, this is considered passed.

from cluecumber.

Also the step counts seem to be correct here.

from cluecumber.

@mpkorstanje this is interesting. What does "the most severe" status mean concretely?

I know the status constants are

PASSED,
SKIPPED,
PENDING,
UNDEFINED,
AMBIGUOUS,
FAILED,
UNUSED

Would that mean that any step status that is lower down the list than any other steps in the scenario determines the scenario status? If so, this is new info. I remember getting my initial info from the Cucumber Slack.

E.g.

Step 1: passed
Step 2: skipped
Step 3: skipped

=> Scenario is skipped?

According to my former info, this would be considered passed.

from cluecumber.

Yes, this specific example the scenario would be skipped.

Not sure what you got from Slack. I do remember this conversation:

https://cucumberbdd.slack.com/archives/C60TKS3SL/p1532961939000185?thread_ts=1532961939.000185&cid=C60TKS3SL

I think you interpreted the comment about scenario no. 3 as me saying the scenario passed. Rather the scenario is skipped, the execution of cucumber is passed (i.e. a zero exit code if this was the only scenario executed).

It is worth nothing that strict mode is default now and non-strict has been removed. So things are much simpler now.

from cluecumber.

Ah cool! So I will refine the state handling again and adjust it to the official way. Thanks for your input!

from cluecumber.

Ready to release in v3.3.0

from cluecumber.