a github action to publish cucumber report on pull request annotations

JavaScript 96.80% Gherkin 3.20%

cucumber-report-annotations-action's People

Contributors

Stargazers

Watchers

Forkers

dheerajgargdelhi121 hfferreira wegel saad-rahman-warsi-cision abhishek00794 audiodaniel kmukund97

cucumber-report-annotations-action's Issues

I can't get de output from the action

The action can read the json file but when i try to get the output it just don't work, here's how i have it

Getting `HttpError: You must authenticate via a GitHub App` during the run

I am trying to get this action working, and getting various errors.
The blocking one is HttpError: You must authenticate via a GitHub App
I tried to use 2 different tokens, both did not work.

This is my action setup:

      - name: Put test run details to GitHub Job annotations
        uses: deblockt/[email protected]
        if: always() && steps.integration-test.outcome != 'skipped'
        with:
          access-token: ${{ secrets.CUCUMBER_ANNOTATIONS_TOKEN }}
          path: "**/cucumber-report.json"

GH job log:

Run deblockt/[email protected]
  with:
    access-token: ***
    path: **/cucumber-report.json
    check-status-on-error: failure
    check-status-on-undefined: success
    check-status-on-pending: success
    annotation-status-on-error: failure
    show-number-of-error-on-check-title: true
    name: Cucumber report

start to read cucumber logs using path **/cucumber-report.json
found cucumber report /home/runner/work/pulse-alp/pulse-alp/integration-test/build/cucumber-report.json
Creating summary:     [3](https://github.com/trilogy-group/pulse-alp/runs/8093899549?check_suite_focus=true#step:15:3) Scenarios (1 failed, 2 passed)
    5[4](https://github.com/trilogy-group/pulse-alp/runs/8093899549?check_suite_focus=true#step:15:4) Steps (1 failed, [5](https://github.com/trilogy-group/pulse-alp/runs/8093899549?check_suite_focus=true#step:15:5) skipped, 48 passed)
Sending cucumber annotations
HttpError: You must authenticate via a GitHub App.
    at /home/runner/work/_actions/deblockt/cucumber-report-annotations-action/v1.9/webpack:/cucumber-report-annotations-action/node_modules/@octokit/request/dist-node/index.js:8[6](https://github.com/trilogy-group/pulse-alp/runs/8093899549?check_suite_focus=true#step:15:6):1
    at processTicksAndRejections (node:internal/process/task_queues:[9](https://github.com/trilogy-group/pulse-alp/runs/8093899549?check_suite_focus=true#step:15:9)6:5)
    at /home/runner/work/_actions/deblockt/cucumber-report-annotations-action/v1.9/webpack:/cucumber-report-annotations-action/index.js:[16](https://github.com/trilogy-group/pulse-alp/runs/8093899549?check_suite_focus=true#step:15:16)4:1

last used token has repo:* and workflow scopes:

Can you please advise what can I fix here?

Schedule workflow not showing the report

Current Behavior

There is no error. Everything happens as it was OK. But then, the Cucumber Report is not shown as it would when triggering the workflow with a commit. My workflow triggers are:
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
schedule:

cron: "0 0 * * *"

stop a build based on the failed test

Hello,

Can i stop/exit a build based on the failed test?

Appreciate your response.

Usage of JobSummary instead of annotations

Hi,

I wanted to run this action from my GH Action as a workflow_dispatch, but found the issue with annotations being assigned to a commit, making it impossible what I had in mind.

Though last year GH released a new feature to expose data in the Job Summary by using an internal API.

It would be awesome if this action could offered an option to display the same data that it now does but writing it into a Job Summary instead (Which would be compatible with workflow_dispatch and other non-pr use cases)

https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

Performance issue when to much error

When a on our cucumber run more than 100 tests fail this action take 40 minutes to run

Fail job on PendingStep

When a step is pending, we should allow to fail the build:

io.cucumber.java.PendingException: Missing event scenario, need to implement it: 'RetrieveCustomerOrder.nominal'
        at com.adeo.sales.customerorder.acceptancetesting.kafka.EventScenario.lambda$of$1(EventScenario.java:111)
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at com.adeo.sales.customerorder.acceptancetesting.kafka.EventScenario.of(EventScenario.java:111)
        at com.adeo.sales.customerorder.acceptancetesting.GenericEventSteps.theSystemHandlesAnEventWithTheFollowingDataUsing(GenericEventSteps.java:78)
        at com.adeo.sales.customerorder.acceptancetesting.GenericEventSteps.theSystemHandlesAnEventWithTheFollowingData(GenericEventSteps.java:72)
        at ✽.the system handles an event RetrieveCustomerOrder with the following data(file:///home/rverjus/Documents/dev/tempo-orchestrator--sources/packages/payment-scheduler-acceptance-testing-cucumber/src/main/resources/features/full_scenario.feature:190)

can have this issue if a step return a PendingException

Upgrade Node 16 (EOL) to Node 20 (TLS)

Node 16 has reached its end of life.

Please upgrade to Node 20:

Modify your actions to run on Node 20 instead of Node 16. For guidance, refer to the Actions configuration settings.

error: RequestError [HttpError]: Resource not accessible by integration

Hi
I have this error trying to use the [email protected]
Do you know what is the problem?

Run deblockt/[email protected]
start to read cucumber logs using path target/cucumber.json
found cucumber report /home/runner/work/ovr-automationTestUI/ovr-automationTestUI/target/cucumber.json
{
scenarioNumber: 1,
failedScenarioNumber: 0,
undefinedScenarioNumber: 0,
pendingScenarioNumber: 0,
succeedScenarioNumber: 1,
stepsNumber: 5,
failedStepsNumber: 0,
skippedStepsNumber: 0,
undefinedStepsNumber: 0,
pendingStepNumber: 0,
succeedStepsNumber: 5
}
1 Scenarios (1 passed)
5 Steps (5 passed)
send global cucumber report data
/home/runner/work/_actions/deblockt/cucumber-report-annotations-action/v1.7/node_modules/@octokit/request/dist-node/index.js:66
const error = new requestError.RequestError(message, status, {
^

RequestError [HttpError]: Resource not accessible by integration
at /home/runner/work/_actions/deblockt/cucumber-report-annotations-action/v1.7/node_modules/@octokit/request/dist-node/index.js:66:23
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async /home/runner/work/_actions/deblockt/cucumber-report-annotations-action/v1.7/index.js:154:9 {
status: 403,
headers: {
'access-control-allow-origin': '*',
'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
connection: 'close',
'content-encoding': 'gzip',
'content-security-policy': "default-src 'none'",
'content-type': 'application/json; charset=utf-8',
date: 'Tue, 25 Jul 2023 11:27:29 GMT',
'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
server: 'GitHub.com',
'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
'transfer-encoding': 'chunked',
vary: 'Accept-Encoding, Accept, X-Requested-With',
'x-content-type-options': 'nosniff',
'x-frame-options': 'deny',
'x-github-api-version-selected': '2022-11-28',
'x-github-media-type': 'github.v3; param=antiope-preview; format=json',
'x-github-request-id': '1480:2484:16A9CC:2DDDB4:64BFB1A1',
'x-ratelimit-limit': '1000',
'x-ratelimit-remaining': '996',
'x-ratelimit-reset': '1690287088',
'x-ratelimit-resource': 'core',
'x-ratelimit-used': '4',
'x-xss-protection': '0'
},
request: {
method: 'POST',
url: 'https://api.github.com/repos/xyuste/ovr-automationTestUI/check-runs',
headers: {
accept: 'application/vnd.github.antiope-preview+json',
'user-agent': 'octokit-core.js/3.1.0 Node.js/16.16.0 (Linux 5.15; x64)',
authorization: 'token [REDACTED]',
'content-type': 'application/json; charset=utf-8'
},
body: '{"name":"Cucumber report","head_sha":"37d666c57f1f7ed2d322dc7977591bc397745c9c","status":"completed","conclusion":"success","output":{"title":"Cucumber report","summary":" 1 Scenarios (1 passed)\n 5 Steps (5 passed)","annotations":[{"path":"test","start_line":0,"end_line":0,"start_column":0,"end_column":0,"annotation_level":"notice","title":"Cucumber report summary","message":" 1 Scenarios (1 passed)\n 5 Steps (5 passed)"}]}}',
request: {
agent: Agent {
_events: [Object: null prototype] {
free: [Function (anonymous)],
newListener: [Function: maybeEnableKeylog]
},
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 443,
protocol: 'https:',
options: [Object: null prototype] { path: null },
requests: [Object: null prototype] {},
sockets: [Object: null prototype] {},
freeSockets: [Object: null prototype] {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
scheduling: 'lifo',
maxTotalSockets: Infinity,
totalSocketCount: 0,
maxCachedSessions: 100,
_sessionCache: {
map: {
'api.github.com:443:::::::::::::::::::::': [Buffer [Uint8Array]]
},
list: [ 'api.github.com:443:::::::::::::::::::::' ]
},
[Symbol(kCapture)]: false
},
hook: [Function: bound bound register]
}
},
documentation_url: 'https://docs.github.com/rest/checks/runs#create-a-check-run'
}

I have the following yml file in the github / workflows
name: Java CI with Maven

on:
workflow_dispatch:
inputs:
tags:
description: Cucumber tags
type: string
default: TEST
required: true

jobs:
build:

permissions:
  actions: read
  contents: read
  security-events: write
  statuses: write

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: Set up JDK 11
  uses: actions/setup-java@v3
  with:
    java-version: '11'
    distribution: 'temurin'
    cache: maven
- name: Build with Maven
  run: mvn clean test -Denvironment=t1 -Dbrowser=remote4 -Dcucumber.filter.tags="@${{ inputs.tags }}"
- name: Upload Cucumber Report
  uses: actions/upload-artifact@v2
  with:
    name: cucumber-report
    path: target/cucumber/reports/cucumber-html-reports

- uses: deblockt/[email protected]
  with:
      access-token: ${{ secrets.GITHUB_TOKEN }}
      path: "target/cucumber.json"

Feature Request: provide output summary

Is it possible to provide us output of these 5 variables so we can pass to subsequent steps ?

   const summarySteps = {
        'failed': globalInformation.failedStepsNumber,
        'undefined': globalInformation.undefinedStepsNumber,
        'skipped': globalInformation.skippedStepsNumber,
        'pending': globalInformation.pendingStepNumber,
        'passed': globalInformation.succeedStepsNumber
    };

Incorrect results are reported

Using deblockt/[email protected]

I get conflicting results with the actual cucumber output compared to what this action is providing in the summary.

Cucumber shows me this:

...
Failing Scenarios:
cucumber -p cwm features/course_unlocker/course_unlocker.feature:10 # Scenario: Instructor wants to unlock the course
cucumber -p cwm features/course_unlocker/course_unlocker.feature:15 # Scenario: Instructor wants to lock the course

2 scenarios (2 failed)
10 steps (10 skipped)

But this action's output:

Run deblockt/[email protected]
  with:
    access-token: ***
    path: **/cucumber-reports.json
    show-global-summary-report: true
    check-status-on-error: failure
    check-status-on-undefined: success
    check-status-on-pending: success
    annotation-status-on-error: failure
    show-number-of-error-on-check-title: true
    number-of-test-error-to-fail-job: -1
    name: Cucumber report
  env:
    SECURITY_FILE_DIRECTORY: files
    BROWSER_HEADLESS: true
...
start to read cucumber logs using path **/cucumber-reports.json
found cucumber report /XXXXXX/results/cucumber-reports.json
Creating summary:     2 Scenarios (2 passed)
    10 Steps (2 failed, 10 skipped)
Sending cucumber annotations
Building all scenario summary
Send core scenario summary

Generally, if I get one pass and one fail, the summary looks correct, as well as if both pass. So it might just be the "all scenarios fail" case that isn't being parsed correctly?

Attaching my json file for reference, in case it's helpful.
cucumber-reports.json

Thanks

UnhandledPromiseRejectionWarning: HttpError: Invalid request.

I'm trying to run this action with json file, but in my tests I get HttpError Invalid requesterror. Would you know what this error would be?

10 Scenarios (1 failed, 9 passed)
    56 Steps (4 failed, 6 skipped, 46 passed)
send global cucumber report data
(node:2201) UnhandledPromiseRejectionWarning: HttpError: Invalid request.

"end_line", "start_line" weren't supplied.
    at /home/runner/work/_actions/deblockt/cucumber-report-annotations-action/v1.7/node_modules/@octokit/request/dist-node/index.js:66:23
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /home/runner/work/_actions/deblockt/cucumber-report-annotations-action/v1.7/index.js:1[54](https://github.com/grafeno-sa/grafeno-pagamentos/runs/5142723681?check_suite_focus=true#step:4:54):9
(node:2201) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2201) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Add support of message format

Actual cucumber JSON format is deprecated, the new officially supported format is "message".

An example file can be found here: cucumber-report-message.json

We should be able to support this format to be compliant we cucumber new features.

Node16

Hi,

I get the following error while using the action:

The following actions uses node12 which is deprecated and will be forced to run on node16: deblockt/[email protected]. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

Handle empty cucumber.json report and not fail workflow run

Test is producing empty cucumber.json

Action

- uses: deblockt/[email protected]
        if: always()
        with:
          name: TEST-REPORT-${{ matrix['tests'] }}
          access-token: ${{ secrets.GITHUB_TOKEN }}
          path: "**/cucumber.json"

Action logs

Run deblockt/[email protected]
  with:
    name: TEST-REPORT-xxx
    access-token: ***
    path: **/cucumber.json
    check-status-on-error: failure
    check-status-on-undefined: success
    check-status-on-pending: success
    annotation-status-on-error: failure
    show-number-of-error-on-check-title: true
start to read cucumber logs using path **/cucumber.json
found cucumber report /runner/_work/xx/xx/xx/xx/build/cucumber/cucumber.json

/runner/_work/_actions/deblockt/cucumber-report-annotations-action/v1.9/webpack:/cucumber-report-annotations-action/reportReader.js:[6](https://github.com/xx/xx/runs/xx?check_suite_focus=true#step:6:6)
        .reduce((a, b) => sum(a, b));
^
TypeError: Reduce of empty array with no initial value
    at Array.reduce (<anonymous>)
    at Object.module.exports.globalInformation (/runner/_work/_actions/deblockt/cucumber-report-annotations-action/v1.9/webpack:/cucumber-report-annotations-action/reportReader.js:6:1)
    at /runner/_work/_actions/deblockt/cucumber-report-annotations-action/v1.9/webpack:/cucumber-report-annotations-action/index.js:[8](https://github.com/xx/xx/runs/xx?check_suite_focus=true#step:6:8)8:1

Expected

Action step doesn't fail

deblockt / cucumber-report-annotations-action Goto Github PK