trivago / cluecumber Goto Github PK

Hi Lax,

There is a bug on counting the number of features passed, Instead of number of features passed its stating the total number of runtime features passed which is basically the total number of scenarios.
PFA the screenshot.

Hi,
I use the plugin with maven,
And I get the error in the title:
Could not render page content: The following has evaluated to null or missing: [ERROR] ==> output [in template "scenario-detail.ftl" at line 123, column 55]

[ERROR] Tip: If the failing expression is known to legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use <#if myOptionalVar??>when-present<#else>when-missing</#if>. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)?? [ERROR] ---- [ERROR] [ERROR] ---- [ERROR] FTL stack trace ("~" means nesting-related): [ERROR] - Failed at: ${output?html} [in template "scenario-detail.ftl" at line 123, column 53] [ERROR] ~ Reached through: #nested [in template "macros/page.ftl" in macro "card" at line 31, column 17] [ERROR] ~ Reached through: @page.card width="12" title="${elemen... [in template "scenario-detail.ftl" at line 35, column 5] [ERROR] ~ Reached through: #nested [in template "macros/page.ftl" in macro "page" at line 14, column 13] [ERROR] ~ Reached through: @page.page base="../.." links=["tag_s... [in template "scenario-detail.ftl" at line 5, column 1] [ERROR] ---- [ERROR] -> [Help 1]
I can't find the file scenario-detail.ftl in my repo or the plugin repo, any help ?
Thanks

Monorepo

Is your feature request related to a problem? Please describe.
It might be possible to have failures in an @afterstep hooks (available since cucumber v3) and this failure may happened after the last step of the scenario/feature.
In this case everything is green (feature, scenario, steps) and no error message is displayed in the cluecumber report. But if you have a look on the basic cucumber html reports, the error message is displayed in red.

Describe the solution you'd like
It would be nice to fail feature + scenario summaries in this case and to write the error message in the scenario details view (like the cucumber html reports).

Describe alternatives you've considered
This is already done for @Before/@after scenario's hooks but it will be different for @afterstep steps hooks because if everything is OK, we won't want to display all the step hook's statuses for every PASSED.

Additional context
N/A

For the purpose of look-and-feel, it’d be nice if there was an included option/switch to toggle capitalization. Long capitalized scenarios and feature names do not look that appealing in my opinion.

Just a suggestion §:-)

Expected behavior

Background steps should display in order they are given

Actual behavior

Background steps display in reverse order in report

Reproduction steps

1. Create and run a feature file that uses Background steps.
2. Open Cluecumber report

Additional details (Maven version, Java version, etc.)

Cluecumber version 0.3.0

Describe the bug
Hello,

After updating to the 1.1.0, I have this error on one of my scenarios:

build 04-Jul-2018 17:42:43 [ERROR] Failed to execute goal com.trivago.rta:cluecumber-report-plugin:1.1.0:reporting (report) on project pamfolio-testing-swing: Could not render page content: An error has occurred when reading existing sub-variable "nameWithArguments"; see cause exception! The type of the containing value was: extended_hash+string (com.trivago.rta.json.pojo.Step wrapped into f.e.b.StringModel)
build 04-Jul-2018 17:42:43 [ERROR]
build 04-Jul-2018 17:42:43 [ERROR] ----
build 04-Jul-2018 17:42:43 [ERROR] FTL stack trace ("~" means nesting-related):
build 04-Jul-2018 17:42:43 [ERROR] - Failed at: #assign stepName = step.nameWithArgum... [in template "scenario-detail.ftl" at line 67, column 33]
build 04-Jul-2018 17:42:43 [ERROR] ~ Reached through: #nested [in template "macros/page.ftl" in macro "card" at line 38, column 17]
build 04-Jul-2018 17:42:43 [ERROR] ~ Reached through: @page.card width="12" title="${elemen... [in template "scenario-detail.ftl" at line 31, column 5]
build 04-Jul-2018 17:42:43 [ERROR] ~ Reached through: #nested [in template "macros/page.ftl" in macro "page" at line 19, column 13]
build 04-Jul-2018 17:42:43 [ERROR] ~ Reached through: @page.page base="../.." links=["featu... [in template "scenario-detail.ftl" at line 5, column 1]
build 04-Jul-2018 17:42:43 [ERROR] ----

To Reproduce
I don't really know how to reproduce this bug because it failed after executing a lot of scenarios and I don't know which one has caused the failure...

Expected behavior
A clear and concise description of what you expected to happen.

Attachments
Unfortunately I cannot send you my target of this build because of security policies.

Additional context
Add any other context about the problem here.

add support to view text in the report which is written by cucumber Scenario.write() method.

Expected behavior

At local system report is generating

Actual behavior

Not able to generate report on docker

Reproduction steps

mvn clean install verify -Pparallel

Additional details (Maven version, Java version, etc.)

Execution report of goal com.trivago.rta:cluecumber-report-plugin:0.2.0:reporting failed: Plugin com.trivago.rta:cluecumber-report-plugin:0.2.0 or one of its dependencies could not be resolved: Failed to collect dependencies at com.trivago.rta:cluecumber-report-plugin:jar:0.2.0 -> com.google.code.gson:gson:jar:2.8.2: Failed to read artifact descriptor for com.google.code.gson:gson:jar:2.8.2: Could not transfer artifact com.google.code.gson:gson:pom:2.8.2 from/to central (https://repo.maven.apache.org/maven2): Connection reset

Reported by @jakubsacha

Describe the bug
Hello,

I have a step in which I pass an interval as a parameter in some examples:

Then the order list should contain 1 order on "" with a weight in interval
Examples:
| var1 | var2 |
| str1 | !]0,100[ |
| str2 | ]0,100[ |
[....]

All tests pass but when the cluecumber-plugin begins to parse the cucumber.json, I get this error:

build 28-Jun-2018 19:27:44 [ERROR] Failed to execute goal com.trivago.rta:cluecumber-report-plugin:1.0.0:reporting (report) on project pamfolio-testing-swing: Could not render page content: Malformed regular expression: Illegal/unsupported escape sequence near index 11
build 28-Jun-2018 19:27:44 [ERROR] \b!]0,100[\b
build 28-Jun-2018 19:27:44 [ERROR] ^
build 28-Jun-2018 19:27:44 [ERROR]
build 28-Jun-2018 19:27:44 [ERROR] ----
build 28-Jun-2018 19:27:44 [ERROR] FTL stack trace ("~" means nesting-related):
build 28-Jun-2018 19:27:44 [ERROR] - Failed at: #assign stepName = stepName?replace("... [in template "scenario-detail.ftl" at line 68, column 37]
build 28-Jun-2018 19:27:44 [ERROR] ~ Reached through: #nested [in template "macros/page.ftl" in macro "card" at line 38, column 17]
build 28-Jun-2018 19:27:44 [ERROR] ~ Reached through: @page.card width="12" title="${elemen... [in template "scenario-detail.ftl" at line 30, column 5]
build 28-Jun-2018 19:27:44 [ERROR] ~ Reached through: #nested [in template "macros/page.ftl" in macro "page" at line 19, column 13]
build 28-Jun-2018 19:27:44 [ERROR] ~ Reached through: @page.page base="../.." links=["featu... [in template "scenario-detail.ftl" at line 5, column 1]

I think this is a regression of cluecumber-plugin:1.0.0 because I had this report well built in 0.8.0.

To Reproduce
Steps to reproduce the behavior:

1. Try to reproduce this error by making a scenario with this kind of sentence and parameters:
   Scenario Outline: Order list weights
   [...]
   Then the order list should contain 1 order on "" with a weight in interval
   Examples:
   | var1 | var2 |
   | str1 | !]0,100[ |
   | str2 | ]0,100[ |
   [...]

Expected behavior
The result parser should not fail even if we pass such parameters with special characters like: !, [, etc....

Attachments
A part of the cucumber.json:

"result": {
"duration": 11685860341,
"status": "passed"
},
"line": 13,
"name": "the order list should contain 1 order on "pf1" with a weight in interval !]0,100[",
"match": {
"arguments": [
{
"val": "1",
"offset": 30
},
{
"val": "pf1",
"offset": 42
},
{
"val": "!]0,100[",
"offset": 78
}
],
"location": "CommonOrderSteps.orderListShouldContain(String,String,String)"
},
"keyword": "Then "
},

Additional context
N/A

charts are displaying well in features and tags but not in scenarios.

Expected behavior

render cucumber Scenario.write() output to @before and @after sections as well

Actual behavior

current it's not rendered there, not sure if a bug :) or its an improvement.

Reproduction steps

run some test with scenario.write() in the @before and @after methods and generate report

Additional details (Maven version, Java version, etc.)

maven 3.5.2
java 1.8

I'm not sure if this is a cluecumber issue, but is there a way to provide documentation to add cluecumber to surefire reports.

@laxersaz
Currently, "doc_string" is displayed below the steps, only if it is failed, and it doesn't have option to hide it.

It would be better, if the report shows the "doc_string" below every steps irrespective of its status and an option to show/hide it.

Thanks,

Expected behavior

Graphics to see errors/success by tags, making easy to see what component is failing

Actual behavior

Right now tags add no value to the report

Hello!
For the example project provided, it says to run mvn clean verify. When I do that on my project and test fail I never receive a report from cluecumber because
"Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.0:test (default-test) on project one: There are test failures."
The .json file is created but I don't think cluecumber is ever "activitated" because it results in a Build Failure. Then it tells me to check look at the surefire-report for the individual test results.
Here is the xml file

4.0.0

<groupId>cucumber-school</groupId>
<artifactId>shouty</artifactId>
<version>0.0.1</version>
<packaging>jar</packaging>
<name>one</name>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>

<dependencies>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-server</artifactId>
        <version>9.2.12.M0</version>
    </dependency>

    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-servlet</artifactId>
        <version>9.2.12.M0</version>
    </dependency>

    <dependency>
        <groupId>com.github.spullara.mustache.java</groupId>
        <artifactId>compiler</artifactId>
        <version>0.9.0</version>
    </dependency>

    <dependency>
        <groupId>oracle</groupId>
        <artifactId>ojdbc6</artifactId>
        <version>11.2.0.3.0</version>
    </dependency>

    <dependency>
        <groupId>info.cukes</groupId>
        <artifactId>cucumber-picocontainer</artifactId>
        <version>1.2.4</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>info.cukes</groupId>
        <artifactId>cucumber-junit</artifactId>
        <version>1.2.4</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.picocontainer</groupId>
        <artifactId>picocontainer</artifactId>
        <version>2.15</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-firefox-driver</artifactId>
        <version>2.46.0</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>1.10.19</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.jsoup</groupId>
        <artifactId>jsoup</artifactId>
        <version>1.7.3</version>
    </dependency>


    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-test</artifactId>
        <version>4.1.7.RELEASE</version>
        <scope>test</scope>
    </dependency>

</dependencies>



<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.3</version>
            <configuration>
                <encoding>UTF-8</encoding>
                <source>1.8</source>
                <target>1.8</target>
                <compilerArgument>-Werror</compilerArgument>
            </configuration>
        </plugin>

        <plugin>
            <groupId>com.trivago.rta</groupId>
            <artifactId>cluecumber-report-plugin</artifactId>
            <version>1.0.0</version>
            <executions>
                <execution>
                    <id>report</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>reporting</goal>
                    </goals>
                    <configuration>
                        <sourceJsonReportDirectory>target/cucumber-report</sourceJsonReportDirectory>
                        <generatedHtmlReportDirectory>target/cluecumber-report</generatedHtmlReportDirectory>
                        <customParameters>
                            <Custom_Parameter>This is a test</Custom_Parameter>
                            <Custom_URL>http://www.google.com</Custom_URL>
                        </customParameters>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Is there a way to make cluecumber run even if there is a build failure?
Thank you!

Expected behavior

Report file should be created when tests failed

Actual behavior

Report file only gets created when the tests pass

Reproduction steps

through the terminal running mvn clean verify test

Additional details (Maven version, Java version, etc.)

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
Java 1.8
<plugin>
                <groupId>com.trivago.rta</groupId>
                <artifactId>cluecumber-report-plugin</artifactId>
                <version>0.7.1</version>
                <executions>
                    <execution>
                        <id>test</id>
                        <phase>test</phase>
                        <goals>
                            <goal>reporting</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <sourceJsonReportDirectory>cucumber-html-reports</sourceJsonReportDirectory>
                    <generatedHtmlReportDirectory>generated-report</generatedHtmlReportDirectory>
                </configuration>
            </plugin>

Expected behavior

Scenario duration should appear as a separate column in the overview page

Integrated to my cucumber project following steps described in https://github.com/trivago/cluecumber-report-plugin/blob/master/README.md

Expected behavior

A sample report should be created at ${project.build.directory}/generated-report

Actual behavior

Exception generated: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.trivago.rta:cluecumber-report-plugin:0.3.0:reporting (default-cli) on project UniteBPCAutomation: The parameters 'generatedHtmlReportDirectory', 'sourceJsonReportDirectory' for goal com.trivago.rta:cluecumber-report-plugin:0.3.0:reporting are missing or invalid

Reproduction steps

As described in README.md

Additional details (Maven version, Java version, etc.)

I moved the configuration block in pom.xml as follows to fix:

<plugin>
    <groupId>com.trivago.rta</groupId>
    <artifactId>cluecumber-report-plugin</artifactId>
    <version>Check the version number above</version>
    <configuration>
           <sourceJsonReportDirectory>${project.build.directory}/cucumber-report</sourceJsonReportDirectory>
            <generatedHtmlReportDirectory>${project.build.directory}/generated-report</generatedHtmlReportDirectory>
    </configuration>
    <executions>
        <execution>
            <id>report</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>reporting</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Step{line=10, name='match $.search == 0', keyword='And ', rows=[]} causes

Java method "com.trivago.rta.json.pojo.Step.returnNameWithArguments()" threw an exception when invoked on com.trivago.rta.json.pojo.Step object

Expected behavior

Introduce additional configuration in Cluecumber report plugin in order to let plugin users change report UI (example : change default background colors, introduce additional header icon, ...)

Actual behavior

Cluecumber report UI cannot be customized "on the fly", unless by overriding HTML / CSS output once processed

Reproduction steps

Not applicable : feature not yet available

Additional details (Maven version, Java version, etc.)

Not applicable : feature not yet available

Expected behavior

Was not expecting there to be any major issues related to encoding with non-english special characters (ie, æøå).

Actual behavior

Looks like the headers for the feature and scenario columns are not being rendered correctly.
None of the steps from the detailed page are affected by this.

I attached a screendump of the the headers from the overview page.

Reproduction steps

1. Make sure language in feature file is set to "no":
   # language: no
2. Use any of the following special characters in the "Feature" and/or "Scenario" section:

Egenskap: Søknadsskjema
  Scenario: En sluttbruker skal kunne fylle ut søknadsskjemaet og søke om foo

Additional details (Maven version, Java version, etc.)

Java version: 1.8.0_152
Apache Maven 3.5.2
Cluecumber 0.2.0

Expected behavior

This is my <Scenario Title>

Actual behavior

This is my

Hi @laxersaz

Problem:
I observe that when a scenario has 'passed' and 'skipped' steps in it, then the scenario is marked as 'Failed' in the report.

Solution:
I think, a scenario should be 'Failed' only when it has 'Undefined' or 'Pending' or 'Failed' steps.
When a scenario has 'Passed' and 'Skipped' steps, the result should be 'Passed'.

Kindly let me know, if my understanding is wrong.

Thanks,

Expected behavior

Scenario Result Chart graph should display on page

Actual behavior

Scenario Result Chart graph is not displayed

Reproduction steps

1. Open Cluecumber report, then click any test scenario link to open Scenario Details page

Additional details (Maven version, Java version, etc.)

Cluecumber version 0.7.0

Related to #65

Expected behavior

Step 2   Then Log in to your account with     | Username | username |    0m 09s 662ms  passed
                                                 | Password | 123456   |                                       
                                                                       

Actual behavior

Step 2 Then Log in to your account with 0m 09s 662ms passed

Reproduction steps

Feature:  Login: Login Test Feature

  Scenario: Login to account
    Given Open Site
    Then Log in to your account with
      | Username | username |
      | Password | 123456      |

https://github.com/fizzed/rocker

Expected behavior

Would it be possible to also filter tables when filtering a chart by type (passed, failed, skipped)?
For example in the Suite Overview when we click on the "passed" type to not display it into the chart, it could be cool to also filter the tables and to display only the "failed" and "skipped" scenarios (so it will be consistent with the Test Suite Result Chart).

Could be the same expected behavior for the Scenario details page to display only the step failure in one click and also useful in the Tag Summary page for reporting.

Actual behavior

We can filter a chart by type but the result in the tables are not filtered.

Reproduction steps

N/A: new feature request

Additional details (Maven version, Java version, etc.)

N/A

• Find out if scenarios should fail on AfterStep failure of the last step?
• Research an official document describing the various step and hook result combinations to consider passed, failed and skipped

https://github.com/cucumber/json-formatter-approval-tests/tree/master/examples

Expected behavior

not sure to post my feature request in the right way, feel free to redirect me to the right medium if available

Do you have any plan to include a feature report on the overview page + feature result drilldown in order to browe results on a feature (or feature + subfeature) basis ?

Actual behavior

No feature based report

Reproduction steps

Not applicable : no feature based report yet

Additional details (Maven version, Java version, etc.)

Not applicable : no feature based report yet

Expected behavior

The scenario should be marked as skipped.

Actual behavior

The scenario is marked as failed.

Reproduction steps

Use the attached JSON file: report.json.zip

The following NullPointerExceptions happens if a scenario won't get executed because of a tag configuration e.g. @ignore which avoids processing in cucumber.

[ERROR] Failed to execute goal com.trivago.rta:cluecumber-report-plugin:0.0.3:reporting (report) on project testsuite-factory: Execution report of goal com.trivago.rta:cluecumber-report-plugin:0.0.3:reporting failed. NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.trivago.rta:cluecumber-report-plugin:0.0.3:reporting (report) on project testsuite-factory: Execution report of goal com.trivago.rta:cluecumber-report-plugin:0.0.3:reporting failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution report of goal com.trivago.rta:cluecumber-report-plugin:0.0.3:reporting failed.
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 20 more
Caused by: java.lang.NullPointerException
at com.trivago.rta.rendering.ReportGenerator.generateReports(ReportGenerator.java:67)
at CluecumberReportPlugin.execute(CluecumberReportPlugin.java:106)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
... 21 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Cucumber json report which leads to the error is attached.

cusumber_json_report.zip

We have separate use cases here:

• Scenarios that should be considered failed when steps are pending (like Cucumber strict mode)
• Scenarios that skip all steps but have passing background steps (currently they are reported as passed)