temyers / cucumber-jvm-parallel-plugin Goto Github PK

I want to use before and after class annotations of junit to execute clean-up actions such as close browser after test. However, the parallel class created by this plugin does not create before and after class in cucumberrunnertest file. I am able to execute clean-up as another cucumber step at the end of my tests but it will be skipped in case of a test failure. So is there a way I can always execute test clean-up regardless of test pass or failure with this plugin?

Hallo,

I think the plugin has a little bug with a great impact on me. All the feature files have to be placed under "/src/test/resources/features/". But I have another folder structure like "/src/test/resources/com/company/test/testcases/". In this case no tests will be found with the given tag and so no tests will run in parallel. So, I have to copy all my existing feature files to "/src/test/resources/features/" to run the tests in parallel. But this is not possible because other code is searching for files in my existing folder structure.

It will be better that all feature files under "/src/test/resources/" will be runnable with this plugin.

THX Martin

I'm seeing that the plug-in is always generating 10 runners even though there are only four feature files.
It looks like it is creating one runner per scenario going by the number of duplicate runners (it is creating the class ok).

I have the plug-in set-up with:
parallelScheme = FEATURE
filterFeaturesByTags = true
tags = ~ignore

It doesn't matter if filterFeaturesByTags = false or tags is empty.

I have tried versions 2.0.0 and 2.0.1

When I using the command 'mvn generate-test-sources' everything is working fine, but when I use the command 'mvn test' the generate classes creates duplicates.

Now it is always using the default character encoding of the system.

I'd like to ask for an enhancement to the plugin. Currently when running tests in parallel on our CI server each generated unit test is being executed regardless of whether it matches filter tags. So, what is happening is as follows:

1. Each test class is executed (so that is 1 per feature). There are about 300+ feature files at present.
2. Each time a test class is executed (via the cucumber junit runner) cucumber scans for features/scenarios that match a tag filter. If none are found it exits else it continues to execute the test.

There is a delay when no matches are found before it exits. Whilst this is not significant for a small number of feature files. The cumulative total of the delay for unnecessarily executing tests is starting to become significant (~12 mins). I suspect the problem lies with cucumber-jvm itself. However, it would be helpful if additional configuration could be added to the plugin whereby some pre-processing of feature files occurs. So, test classes will only be generated for those that match filter tags that have been supplied.

Would be cool if there were some way to push the concurrency of the cucumber features even more. And one way to do this is to split up a feature file into smaller parts.

Take the following feature

Feature: Upload photos

  As a user,
  I should be able to upload my photos

  Background:
    Given I have an account
    And I have a couple of photos on my device
    And I have logged into my account

  Scenario Outline: As a user I should be able to upload one or more photos
    When I select <num> photos for upload
    Then I expect <photos> photos to appear as uploaded
    And I expect <events> events to appear in the event view

  Examples:
    | num | photos | events |
    |   1 |      1 |      1 |
    |   4 |      4 |      1 |

  Scenario: As a user I should not be able to upload duplicate photos
    When I upload the same photo twice
    Then I expect 1 photos to appear as uploaded
    And I expect 1 events to appear in the event view

If i'm to run this scenario then this would take quite a while as it really is comprised of 3 test runs.

• one for "As a user i should be able to upload one or more photos" with the 1, 1 ,1 example
• then one for the 4,4,1 example
• and lastly "As a user I should not be able to upload duplicate photos"

The way this could be improved was if this plugin could create three feature files from this single file where the feature file "1.feature" would contain:

Feature: Upload photos

  As a user,
  I should be able to upload my photos

  Background:
    Given I have an account
    And I have a couple of photos on my device
    And I have logged into my account

  Scenario Outline: As a user I should be able to upload one or more photos
    When I select <num> photos for upload
    Then I expect <photos> photos to appear as uploaded
    And I expect <events> events to appear in the event view

  Examples:
    | num | photos | events |
    |   1 |      1 |      1 |

Then the next "2.feature" file contains

Feature: Upload photos

  As a user,
  I should be able to upload my photos

  Background:
    Given I have an account
    And I have a couple of photos on my device
    And I have logged into my account

  Scenario Outline: As a user I should be able to upload one or more photos
    When I select <num> photos for upload
    Then I expect <photos> photos to appear as uploaded
    And I expect <events> events to appear in the event view

  Examples:
    | num | photos | events |
    |   4 |      4 |      1 |

And then lastly "3.feature" contains

Feature: Upload photos

  As a user,
  I should be able to upload my photos

  Background:
    Given I have an account
    And I have a couple of photos on my device
    And I have logged into my account


  Scenario: As a user I should not be able to upload duplicate photos
    When I upload the same photo twice
    Then I expect 1 photos to appear as uploaded
    And I expect 1 events to appear in the event view

And then finally we'd use the existing functionality of the plugin to create dedicated test runners for each feature file.

WDYT? Too tall an order? or is it doable? Heck, maybe it would be better to implement this directly into cucumber.

Hi all,

Basically problem is described in title:
As far as I know there are 2 ways to add listeners to TestNG tests: 1. Using annotations in runner (which is not an option cuz we're not able to customize runners)
2. Using testng.xml - and in this case we're required to provide package or location of test classes using package hierarchy.

As I understand there is no way to add listeners when running testNG?

I have a Cucumber/Selenium framework and have each test designed to extend a class in order to initialize the web driver upon suite execution and then subsequently destroy the web driver after suite execution.

Is there any way to get the generated runner classes to extend a class?
Any way to inject code into the body of the generated runner class, in order to add @BeforeClass and @afterclass hooks?

I understand that my overall design may need to change in order to support parallelization with Selenium,
but was hoping there'd be a way to manipulate the generated runner class.

Thank you.

I am not sure if this is a bug or a setting I am missing but my browsers remain open until all runners complete their tests. I end up having 20+ open browsers. Is there any setting to setup the browser to close after each runner complete the task?

This was especially visible when I used <parallelScheme>SCENARIO</parallelScheme>

The cucumber-jvm-parallel-plugin allows arbitrary code injection via tags element of the configuration.

<execution>
	<id>generateRunners</id>
	<phase>validate</phase>
	<goals><goal>generateRunners</goal></goals>
	<configuration>
	<glue>com.(redact).automation</glue>
	<outputDirectory>${project.build.directory}/generated-test-src/cuke</outputDirectory>
	<featuresDirectory>src/test/resources/features/</featuresDirectory>
	<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
	<format>junit,json</format>
	<monochrome>true</monochrome>
	<tags>**CODE-TO-INJECT**</tags>
	<filterFeaturesByTags>true</filterFeaturesByTags>
	<namingScheme>simple</namingScheme>
</configuration>

The contents of the tags element is inserted directly into the generated code AS-IS and un-sanitised.
The Arbitrary string could include code. The tags should be sanitised.

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
    features = {"classpath:features/LandingPage.feature"},
    format = {"junit:target/cucumber-parallel/1.junit", "json:target/cucumber-parallel/1.json", "pretty"},
    monochrome = true,
    tags = {**CODE-TO-INJECT**},
    glue = { "com.(redact).automation" })
public class Parallel01IT {
}

How do I generate runners with no tags?

I have found that if I set the following config:

                    <configuration>
                        <tags></tags>

or

                    <configuration>
                        <tags>  </tags>

Then cucumber-jvm-parallel-plugin generates runners like:

@CucumberOptions(
    tags = {"@complete", "@accepted"},

However, if I use the following command-line property, it does seem to work:

mvn -Dcucumber.tags= test-compile

I am using ver 1.2.1

At the moment the json report outputs to cucumber-parallel. This should be made configurable

@temyers
in the pom.xml i have not added

                <executions>
                    <execution>
                        <id>generateRunners</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>generateRunners</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>src/test/java/output</outputDirectory>
                            <glue>com.test.cucumber.stepdefinitions</glue>
                            <featuresDirectory>src/test/resources/features/</featuresDirectory>
                            <cucumberOutputDir>target</cucumberOutputDir>
                            <format>json,html</format>
                        </configuration>
                    </execution>
                </executions>

when i run mvn clean validate, i get @completed and @Accepted tag in the Parallel01IT class

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
    features = {"classpath:features/Sample.feature"},
    format = {"json:target/1.json", "html:target/1.html", "pretty"},
    monochrome = false,
    tags = {"@complete", "@accepted"},
    plugin = {"com.cucumber.listener.ExtentCucumberFormatter:"},
    glue = { "com.test.cucumber.stepdefinitions" })
public class Parallel01IT {
}

Can you please help me on this?

The configuration option of tags complicates custom run configurations in maven.

so say i have the property "cucumber.tags", which is injected into the tags configuration of the plugin and i'd like to run my project with mvn -Dcucumber.tags="@completed", "~@ignored" then this doesn't work properly.

I've also tried to use singular quotes around the whole expression and escaping the double quotes. The result always seems to be that the last double quote is consumed.

Ideally i'd like to be able to run with mvn -Dcucumber.tags=@completed,~@ignored where the plugin then correctly transforms this into "@completed","~@ignored" in when creating the runners.

In a large project, if you only want to run small number of features with a specific tag, classes are still generated for all feature files.

When run in parallel, each Test case has a setup/tear down time which can lead to significant overhead.

Add support for filtering the generated output files to only those features that contain the tags to run.

#53 (comment)

In the Parallel01IT.java i wouls want to include plugin ={"com.cucumber.listener.ExtentCucumberFormatter:"}

Ideally the Runner would look like

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
    features = {"classpath:features/MyFeature.feature"},
    format = {"json:target//1.json", "html:target//1.html", "pretty"},
    monochrome = false,
    tags = {"@accepted"},
    plugin = {"com.cucumber.listener.ExtentCucumberFormatter:"},
    glue = { "com.cucumber.stepdefinitions" })
public class Parallel01IT {
}

how can i add this ?

Hi! I've experienced issue with screenshots display in the Cucumber-JVM-Reports:
if there are 2 failed tests in a suite, 1 screenshot for each test and screenshots look the same visually and a size is the same, then link to the 2nd screenshot doesn't work: link to the 2nd screenshot expands/collapses the 1st screenshot.


Is it a feature or a bug?

Hi,

Cucumber Parallel plugin causing compilation issues on windows while execution.
Parallel java files created during execution are showing path in wrong format. forward slash '/' should be used in specifying the path instead of the backward slash for the format attribute.

format = {"json:D:\MNSProject\test\shopping-test-framework\shopping-tests-api-ls/target/cucumber-reports/1.json", "pretty"},

This throws illegal escape character errors.

You should update the readme to say 1.1.2

pom.xml

 <parallelScheme>FEATURE</parallelScheme>
 <tags>"@google"</tags>
 <filterFeaturesByTags>true</filterFeaturesByTags>

There is only one feature file has tag @google. but runner classes are generated for non matching or no tags on features file. which is unexpected.

Feature with only matching tag should get filtered.

** Edited
sorry it was mistake from my side. Configuration property

<filterFeaturesByTags>true</filterFeaturesByTags> must be true with 
 <parallelScheme>FEATURE</parallelScheme>
 <tags>"@google"</tags>

Still if we use following combination runner should not generate runner for all other non matching or empty Feature tag match.

<filterFeaturesByTags>false</filterFeaturesByTags>
 <parallelScheme>FEATURE</parallelScheme>
 <tags>"@google"</tags>

Even though I have this:

<includes>
    <include>**/*IT.class</include>
</includes>

The Tests are not being found. Can we just use the "Test.java" suffix instead? I can make a pull request for this, but please confirm the work allowed

At the moment, users can choose between junit and testng formats, which changes the runner-template.vm file used to generate the java.

We have the requirement for our runner file to have some additional annotations (eg the test classes extend from a BaseTest class), and so I've forked this project and implemented a custom template.
(fork is here: https://github.com/Frameworkium/cucumber-jvm-parallel-plugin )

I've tried to create a pull request that allows users to specify a parameter in their pom - alongside the standard parameters for the plugin - to a vm file in their project. For example, it might be in their /resources/ folder: /resources/custom-runner.vm

However, this doesn't work, as Velocity is unable to find the file on the class path (as presumably it's still looking in it's /resources/ folder?)

What do I need to add to make this work, or could a new feature be implemented to enable this?

on linux, Parallel01IT is created for feature2.feature
on Windows, Parallel01IT is created for feature1.feature

I would like to run the 1.3.0-SNAPSHOT version of the plugin, do you have a repository for development builds?

I tried using jitpack for this, however this failed:

jitpack/jitpack.io#590

Hi,

I've started to work on a small enhancement that would use the feature file names to generate classes with that name (camelCased), as the current behavior generates 1 class per feature file, but with a generic name.

This is the main change :
vincent-fuchs@c3590d3

The problem is that currently it breaks most of the existing tests, since the output file is different.

Do you think this could be useful ? maybe we can add a flag in the config to enable that feature (disabled by default), so that it doesn't impact existing users...

If you're ready to take that feature, I'll fix the tests in my PR, then submit.

Hi, Is there way to generate single rerun.txt file after the parallel tests are completed so we can just rerun all failed tests as post single thread instead rerun it in parallel mode. For example. currently when we run parallel tests using this plugin, we don't have the capability to add "rerun:rerun.txt" plugin to all the runners. I know that this pull request is open related to rerun functionality. Please kindly let us know if this pull request will fix our issue. Btw thanks for building this plugin, It's been helping us everyday with our test execution.

Thanks
Mohammad

In order to support contributions,
As the project owner
I want a consistent codeing style to make merge/review easier.

Currently there is no way to set java package for runners.

HI,

for browser compatibility testing using selenium grid i need to pass different parameter for each generate test.

Would you let me know how can i achieve this ?

Our project has many features directories, and in addition there are some nested features directories, so under /features/ there are the names of all of the subdirectories and within them there are feature files as well as directories containing feature files. Is making cucumber-jvm-parallel-plugin more flexible in this direction something that you would consider taking on, or should we just put together an implementation and submit a pull request?

Hey @temyers,

I tried new code you merged, Its good in compact have great functionality, however there is issue while generating jUnit Runner class.

1. Pass tags in pom <tags>"@google"</tags>
2. Use property <parallelScheme>SCENARIO</parallelScheme>
3. mvn clean verify.

Expected :
In feature property if feature file path followed by line number as follow
features = {"D:/Test/resources/features/Test.feature:**40**"}
runner class should not include
tags = {"@google"},

Hello,
Version 1.2.1. I got errors in generated Java code, when generating names using option namingScheme='feature-title' features, for files like "some.test.feature" The '.' is not a valid character in a Java class name.
Here is what I would like to achieve:

• Given I have a feature file with '.' inside a file name
• And I use option namingScheme='feature-title'
• When I want to generate a runner using cucumber-jvm-parallel-plugin
• Then I get a runner with a class name, where '.' characters are replaced with a character, that can be used inside a Java class name

I have my feature files in src/it/resources, as per the Maven standard layout

If I configure cucumber-jvm-parallel-plugin like:

                    <configuration>
                        <!-- The diectory, which must be in the root of the runtime classpath, containing your feature files.  -->
                        <featuresDirectory>src/it/resources</featuresDirectory>

... then it finds all my feature files, but generates invalid runner files which look like:

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
    features = {"classpath:resources/uk/co/me/my-feature.feature"},
    format = {"json:target/cucumber-parallel/3.json", "pretty"},
    monochrome = true,
    tags = {},
    glue = { "uk.co.me" })
public class Myfeature03IT {

Which fails at runtime because the feature path should be "classpath:uk/co/me/my-feature.feature", not "classpath:resources/uk/co/me/my-feature.feature"

Do you know how I can fix this?

TIA

Hi,

I setup the plug-in as described @ https://github.com/temyers/cucumber-jvm-parallel-plugin.
However, I am facing following error, when trying maven build. The issue is same with v1.2.0, 1.2.1 and 1.3.0 as well.

Failed to execute goal com.github.temyers:cucumber-jvm-parallel-plugin:1.3.0:generateRunners (generateRunners) on project colspace: Execution generateRunners of goal com.github.temyers:cucumber-jvm-parallel-plugin:1.3.0:generateRunners failed: A required class was missing while executing com.github.temyers:cucumber-jvm-parallel-plugin:1.3.0:generateRunners: org/apache/velocity/context/Context

Please help!!

Hey @temyers ,

Are there any possibilities to generate runners per tags in feature?
That will be another workaround of splitting scenarios on parts.

Like:

  Scenario Outline: func test
    Given some <data>
    And test func

  @theTagPart2
    Examples:
    |data|
    |any|

  @theTagPart1
    Examples:
    |data|
    |some|

As result we will have 2 runners with 2 different tags.

The plugin doesn't work with the xml format described in the readme, namely:

<plugin>
...
<executions>
...
<configuration>...</configuration>
</executions>
</plugin>

But I find that this works:

<plugin>
...
<configuration>...</configuration>
<executions>...</executions>
</plugin>

Added following comment from @bhattankit in issue#15

This could be a good addition. Can we also add functionality to add suffix to the generated class. The reason for this is that we have a couple maven profile that isolates the kind of tests to run.

hi,

i'm in a situation where i have static webdriver because of specific design (not using cucumber dependency injection). i'm experiencing problems with parallel execution (still under investigation) and my first worry is about the static web driver.
i know that using the cucumber-jvm-parallel-plugin , it uses surefire forks(JVMs) instead of threads which means separate JVM for each feature file BUT it is configured to reuse the forks which may prove problematic with static variables. is it true that its not possible to set reuseForks=false ? (will be ignored by the plugin?)
not reusing the JVM forks will ensure full separation in parallel execution in the current design I have.

Thanks

If you use custom report formatter like

<configuration>
  ...
  <format>com.some.Formatter</format>
  ...
</configuration>

It causes generated classes look like this:

@RunWith(Cucumber.class)
@CucumberOptions(
plugin = {"com.some.Formatter:target/cucumber-parallel/1.com.some.Formatter"},
public class Parallel01IT {
}

If there are failures in the integration tests, the TravisCI build should be marked as failed.

@temyers is there a way we can re-run failed scenarios?

When run on Windows, feature locations include '' characters, rather than '/'

@temyers

i have my feature files inside multiple directory as below /src/test/resources/com/automationrhapsody/cucumber/parallel/tests/wikipedia

in my pom.xml if i specify the entire path it does not work as expected

                     <configuration>
                         <glue>com.automationrhapsody.cucumber.parallel.tests.wikipedia</glue>
                          <featuresDirectory>src/test/resources/com/automationrhapsody/cucumber/parallel/tests/wikipedia</featuresDirectory>
                            <cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
                            <format>json,html</format>
                            <tags>"~@ignored"</tags>
                        </configuration>

Genrerated Runner class is

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
    features = {"classpath:wikipedia/main-page.feature"},
    format = {"json:target/cucumber-parallel/2.json", "html:target/cucumber-parallel/2.html", "pretty"},
    monochrome = false,
    tags = {"~@ignored"},
    glue = { "com.automationrhapsody.cucumber.parallel.tests.wikipedia" })
public class Parallel02IT {
}

classpath is in-correct it should have been src/test/resources/com/automationrhapsody/cucumber/parallel/tests/wikipedia/main-page.feature

I get the following error when attempting to use 1.2.0

[ERROR] Failed to execute goal com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0:generateRunners (generateRunners) on project test-project: Execution generateRunners of goal com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0:generateRunners failed: A required class was missing while executing com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0:generateRunners: com/google/common/base/CaseFormat
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/C:/Users/kthomson/.m2/repository/com/github/temyers/cucumber-jvm-parallel-plugin/1.2.0/cucumber-jvm-parallel-plugin-1.2.0.jar
[ERROR] urls[1] = file:/C:/Users/kthomson/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar
[ERROR] urls[2] = file:/C:/Users/kthomson/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar
[ERROR] urls[3] = file:/C:/Users/kthomson/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
[ERROR] urls[4] = file:/C:/Users/kthomson/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar
[ERROR] urls[5] = file:/C:/Users/kthomson/.m2/repository/org/codehaus/plexus/plexus-utils/1.1/plexus-utils-1.1.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[maven.api, parent: null]]
[ERROR]
[ERROR] -----------------------------------------------------: com.google.common.base.CaseFormat

I use testng, having a xml test suite.

the test suite have several feature, each feature have it's test. just like this:

<suite name="regression_suite" verbose="1" parallel="false" thread-count="3"> <test name="regression1"> <parameter name="browser" value="ie"/> <parameter name="browser_version" value="8" /> <classes> <class name="F00001"/> </classes> </test> <test name="regression2"> <parameter name="browser" value="ie"/> <parameter name="browser_version" value="9" /> <classes> <class name="F00002"/> </classes> </test> <test name="regression3"> <parameter name="browser" value="ie"/> <parameter name="browser_version" value="10" /> <classes> <class name="F00003"/> </classes> </test> </suite>

sometimes the cases (F00003) running in the end would fail and the log shows glue code was not found while the cases running in the beginning with the same glue code pass

There should be a fature to set a custom name on the json report at the CucumberOptions plugin. Otherwise the reports are named with the fork number and it's not easy to identify the remote device where the test was actually executed.
Would it be possible to rename it with the device platform and browser getting them thorugh system or env properties?

I'm happy to help on this if someone provides some guidance

Given the configuration:

            <plugin>
                <groupId>com.github.temyers</groupId>
                <artifactId>cucumber-jvm-parallel-plugin</artifactId>
                <version>1.2.1</version>
                <executions>
                    <execution>
                        <id>generateAndroidRunners</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>generateRunners</goal>
                        </goals>
                        <configuration>
                        <configuration>
                            ...
                            <tags>"@android"</tags>
                            <namingScheme>simple</namingScheme>
                        </configuration>
                        </configuration>
                    </execution>
                    <execution>
                        <id>generateIOSRunners</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>generateRunners</goal>
                        </goals>
                        <configuration>
                            ...
                            <tags>"@ios"</tags>
                            <namingScheme>simple</namingScheme>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

If i have a single feature in the project this configuration would only create Parallel01IT.java, where i would expect this to create two files, one with the IOS configuration one with the Android configuration.

However the workaround for now is to have the naming scheme on one of these executions set to feature-title.

I guess something like #19 would be a solution for this, or a naming scheme which is a bit more intelligent.