stryker-mutator / azure-devops-mutationreport-publisher Goto Github PK

I'm attempting to use this plugin on our Azure devops server but the report tab seems to be infinitely loading when I open it.
I have included my Yaml pipeline (Stryker is running with correct output and the file is generated) and some logs from the Chrome Developers Console. This last one seems to contain an error.

Versions

• dotnet-stryker: 0.20.0
• azure-devops-mutationreport-publisher: 0.1.3

Yaml

- task: DotNetCoreCLI@2
    displayName: 'Install dotnet-stryker'
    inputs:
      command: custom
      custom: tool
      arguments: update dotnet-stryker --tool-path $(Agent.ToolsDirectory)

  - task: Powershell@2
    displayName: 'Run mutation test on ${{ parameters.testName}}'
    inputs:
      workingDirectory: ${{ parameters.workingDirectory }}
      targetType: 'inline'
      script: $(Agent.ToolsDirectory)/dotnet-stryker ${{ parameters.strykerCommands}}

  - task: PublishMutationReport@0
    displayName: 'Publish Mutation Test Report'
    inputs:
      reportPattern: '**/mutation-report.html'

Developer console

I can confirm, by using the diagnostic logs, that the html file is found and uploaded by the task.

Task icon is not working, icon is not displayed

At the moment it seems to default display report names to mutation-report-(x).html. It should use the same report name that it was originally generated as (e.g. name assigned in the stryker-config.json)

Describe the bug
We get sometimes errors in Azure Devops using Stryker with the error message:
Mutation Report Publisher failed to load. No HTML report found..

When we run the pipeline 5 times, this happens 4 times. 1 of the 5 times, we get an Report shown up in DevOps,
There are no changes on the code
We implement an YAML pipeline with multiple reports.

Can you assist on those error(s)?
Correct Result:

Error:

Pipeline:

    - task: DotNetCoreCLI@2
      displayName: 'Install dotnet-stryker'
      inputs:
        command: custom
        custom: tool
        arguments: 'install dotnet-stryker --tool-path $(Agent.BuildDirectory)/tools'

    - task: PowerShell@2
      displayName: 'dotnet tool update dotnet-stryker'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\'
        targetType: inline
        script: dotnet tool update dotnet-stryker

    - task: PowerShell@2
      displayName: 'Run dotnet-stryker for xxxxApi.Tests'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\xxxxApi.Tests'
        targetType: inline
        script: dotnet stryker --config-file "../stryker-config.json"

    - task: PowerShell@2
      displayName: 'Run dotnet-stryker for xxx.Business.Tests'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\xxx.Business.Tests'
        targetType: inline
        script: dotnet stryker --config-file "../stryker-config.json"

    - task: PowerShell@2
      displayName: 'Run dotnet-stryker for xxx.DataAccess.Tests'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\xxxxxxxxx.Tests'
        targetType: inline
        script: dotnet stryker --config-file "../stryker-config.json" --mutate "**/*Repository.cs"

    - task: PublishMutationReport@1
      displayName: 'Publish Mutation Test Report'
      inputs:
        reportPattern: '**/mutation-report.html'

Config json file:

{
  "stryker-config": {
    "mutation-level": "Standard",
    "reporters": [ "html", "progress" ],
    "thresholds": {
      "high": 80,
      "low": 65,
      "break": 50
    }
  }
}

hi guys,

Do you know if this extension will be released any time soon?
What is the status? am I able to use it to see if it works or this is very preliminary version?

Thanks,

Greg

Types have been added in #56

Vulnerable Library - azure-pipelines-task-lib-3.3.1.tgz

Path to dependency file: /extension/PublishMutationReport/package.json

Path to vulnerable library: /extension/PublishMutationReport/node_modules/minimatch/package.json

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (azure-pipelines-task-lib version)	Remediation Available
CVE-2022-37614	High	9.8	mockery-1.7.0.tgz	Transitive	N/A*	❌
CVE-2022-3517	High	7.5	minimatch-3.0.4.tgz	Transitive	N/A*	❌

*For some transitive vulnerabilities, there is no version of direct dependency with a fix. Check the section "Details" below to see if there is a version of transitive dependency where vulnerability is fixed.

Details

CVE-2022-37614

Vulnerable Library - mockery-1.7.0.tgz

Simplifying the use of mocks with Node.js

Library home page: https://registry.npmjs.org/mockery/-/mockery-1.7.0.tgz

Path to dependency file: /extension/PublishMutationReport/package.json

Path to vulnerable library: /extension/PublishMutationReport/node_modules/mockery/package.json

Dependency Hierarchy:

• azure-pipelines-task-lib-3.3.1.tgz (Root Library)
  • ❌ mockery-1.7.0.tgz (Vulnerable Library)

Found in base branch: master

Vulnerability Details

Prototype pollution vulnerability in function enable in mockery.js in mfncooper mockery commit 822f0566fd6d72af8c943ae5ca2aa92e516aa2cf via the key variable in mockery.js.

Publish Date: 2022-10-12

URL: CVE-2022-37614

CVSS 3 Score Details (9.8)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: High
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2022-37614

Release Date: 2022-10-12

Fix Resolution: mockery - 2.1.0

Step up your Open Source Security Game with Mend here

CVE-2022-3517

Vulnerable Library - minimatch-3.0.4.tgz

a glob matcher in javascript

Library home page: https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz

Path to dependency file: /extension/PublishMutationReport/package.json

Path to vulnerable library: /extension/PublishMutationReport/node_modules/minimatch/package.json

Dependency Hierarchy:

• azure-pipelines-task-lib-3.3.1.tgz (Root Library)
  • shelljs-0.8.5.tgz
    • glob-7.1.6.tgz
      • ❌ minimatch-3.0.4.tgz (Vulnerable Library)

Found in base branch: master

Vulnerability Details

A vulnerability was found in the minimatch package. This flaw allows a Regular Expression Denial of Service (ReDoS) when calling the braceExpand function with specific arguments, resulting in a Denial of Service.

Publish Date: 2022-10-17

URL: CVE-2022-3517

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Release Date: 2022-10-17

Fix Resolution: minimatch - 3.0.5

Step up your Open Source Security Game with Mend here

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency copy-webpack-plugin to v12
• chore(deps): update dependency css-loader to v7
• chore(deps): update dependency lerna to v8
• chore(deps): update dependency sass-loader to v14
• chore(deps): update dependency style-loader to v4
• chore(deps): update dependency typescript to v5
• chore(deps): update dependency webpack-dev-server to v5
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• fix(deps): update react monorepo to v18 (major) (react, react-dom)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Using https://github.com/microsoft/azure-devops-extension-hot-reload-and-debug

We need to provide already setup development extension configuration and a complete how-to setup your development environment if we ever expect contributions.

The mutationreport publisher right now works by publishing the HTML file and loading that in an IFrame (iframe in iframe), instead of publishing the JSON of the mutation testing report itself.

I have only communicate my dissatisfaction verbally, and did a poor job at it. Sorry for that. Let me try to convince you with arguments in this issue.

• It only works for single-file reports.
  • Both Stryker and Stryker4s spread reports out over 3 files. Which means they have to change, or are not supported.
  • You force other frameworks which want to align on our mutation testing report JSON and HTML report to use a single file report. This is pretty hostile IMO. You might want to load other external resources on your html page, you now force all those resources to be embedded. Another scenario is that they might only want to support our json format, and are not be ready to support our HTML reporter. Too bad, you will not be supported.
  • It goes against the HTML specification and way of working.
  • We're forcing us to limit to 1 html file in the future. If we would ever want add features like lazy loading parts of the report, we're unable to do so without breaking the azure devops publisher.
• It's a generic HTML file publisher.
  • This might sound like a lame excuse, but I like to call things what they are. Why not call it a html-file-publisher instead?
  • Since the publisher doesn't know anything about the mutation report, it cannot add logic later. Something like inline annotations comes to mind: (for example, how eslint does it in github: https://github.com/hallee/eslint-action/blob/master/screenshots/annotation.png), or add a summary of the mutation score somewhere. This won't be possible to add these kind of features if we don't have the data.
• Maintainability.
  • I don't think loading an iframe in an iframe is great for maintainability. What if Microsoft decides to not allow iframes inside their iframe?
  • We're dependent on an external dependency (i.e. iframe resizer), so we need to maintain that dependency.
  • We're integrating 3 things (azure devops, mutation testing elements and the resizer) into one page, it's just a matter of time before this breaks IMO. Creating e2e tests for it which test the integration will help us with that, but those are also expensive to create and maintain.

My proposal is to create a new publisher that works with the report JSON and rebrand this one as a, more generic, html-file publisher.

@hugo-vrijswijk @richardwerkman @simondel @Mobrockers What are your thoughts?

Azure DevOps is beginning the process to deprecate the Node execution handler in favor of Node10. As of a few days ago, custom tasks across the board spam warnings into build pipelines, the mutation publishing task being one of them.

See here for more info:
https://aka.ms/migrateTaskNode10

Environment
Azure Devops Server 2020 update 1.2
Mutation Report Publisher 1.2.0

Describe the bug
The mutation report fail to load since the update 1.2 of the task on an azure devops server 2020 update 1.2 on premise deployment.
When opening the tab, this url is called : http://{base_url}/DefaultCollection/{team_project_id}/_apis/build/builds/{build_id}/attachments/stryker-mutator.mutation-report
An error 400 occurs when accessing the report tab with the message : The requested REST API version of 7.2 is out of range for this server. The latest REST API version this server supports is 6.1.
This blocks the loading of the tab.

The publication seems to works just fine, it's just the tab's display that is impacted.

We can use https://github.com/maciejmaciejewski/azure-pipelines-protractor for inspiration (@robertlyson thanks so much for finding this)

The README of this repo is empty right now and I also can't find the extension on the marketplace. Could you fill the README with the state of the project and how to use it?

Hi, followed the instructions as outlined on the blog and installed the plug in but my build is reporting 0 artifacts

Have a run stryker task

steps:

• task: DotNetCoreCLI@2
  displayName: 'Stryker - Project'
  inputs:
  command: custom
  custom: 'stryker '
  arguments: '-im [''GetHashCode''] -r "[''html'']" -th 90 -tl 80 -tb 50'
  workingDirectory: 'project.Test'

This works and logs as expected

Second task
steps:

• task: stryker-mutator.mutation-report-publisher.44d9cfb7-7efd-48e2-b2ae-4750950271be.PublishMutationReport@0
  displayName: 'Publish Mutation Test Report'
  inputs:
  reportPattern: 'project.Test/**/*.html'

The publish tasks completes without error but no artifact is created and no extra tab displays.
Do i also need to publish this artifact as a final step

Thanks

Snyk is giving high severity issues on azure-pipelines-task-lib. Could you upgrade this package?

Issues to fix by upgrading:
Upgrade [email protected] to [email protected] to fix
✗ Prototype Pollution [High Severity][https://security.snyk.io/vuln/SNYK-JS-MOCKERY-3043117] in [email protected]
introduced by [email protected] > [email protected]

Issues with no direct upgrade or patch:
✗ Missing Release of Resource after Effective Lifetime [High Severity][https://security.snyk.io/vuln/SNYK-JS-INFLIGHT-6095116] in [email protected]
introduced by [email protected] > [email protected] > [email protected] > [email protected]
No upgrade or patch available

✗ Regular Expression Denial of Service (ReDoS) [High Severity][https://security.snyk.io/vuln/SNYK-JS-SEMVER-3247795] in [email protected]
introduced by [email protected] > [email protected]
This issue was fixed in versions: 5.7.2, 6.3.1, 7.5.2

Vulnerable Library - azure-pipelines-task-lib-3.4.0.tgz

Path to dependency file: /extension/PublishMutationReport/package.json

Path to vulnerable library: /extension/PublishMutationReport/node_modules/minimatch/package.json

Found in HEAD commit: 37cd0991258d7e02a06f942fd79377f0906b1e89

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (azure-pipelines-task-lib version)	Remediation Available
CVE-2022-3517	High	7.5	minimatch-3.0.4.tgz	Transitive	N/A*	❌

*For some transitive vulnerabilities, there is no version of direct dependency with a fix. Check the section "Details" below to see if there is a version of transitive dependency where vulnerability is fixed.

Details

CVE-2022-3517

Vulnerable Library - minimatch-3.0.4.tgz

a glob matcher in javascript

Library home page: https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz

Path to dependency file: /extension/PublishMutationReport/package.json

Path to vulnerable library: /extension/PublishMutationReport/node_modules/minimatch/package.json

Dependency Hierarchy:

• azure-pipelines-task-lib-3.4.0.tgz (Root Library)
  • shelljs-0.8.5.tgz
    • glob-7.1.6.tgz
      • ❌ minimatch-3.0.4.tgz (Vulnerable Library)

Found in HEAD commit: 37cd0991258d7e02a06f942fd79377f0906b1e89

Found in base branch: master

Vulnerability Details

A vulnerability was found in the minimatch package. This flaw allows a Regular Expression Denial of Service (ReDoS) when calling the braceExpand function with specific arguments, resulting in a Denial of Service.

Publish Date: 2022-10-17

URL: CVE-2022-3517

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Release Date: 2022-10-17

Fix Resolution: minimatch - 3.0.5

Step up your Open Source Security Game with Mend here