alexneo2003 / playwright-azure-reporter Goto Github PK

Hi,
When i setup my project with all the details i am getting this No Test Point found Error. I have double checked my Test Plan ID and Test Case ID they are correct i am not sure why i am getting this error. Can anyone pls help me to understand what i am doing wrong?

I have removed the token and omitted some data while taking the Screenshot. but while running all the data was there.

@alexneo2003

getting this error

my TFS test case status is as below

could you please help me with what could be wrong

I have a need to run tests not only on different browsers, but also on different operating systems. Hence I made a configuration in Azure DevOps for each relevant combo.
So my question is : has anyone found a clean way to correctly link bi-dimentional configurations using testPointMapper ?

For example : Windows + Firefox / Windows + Chrome / Android + Chrome / Mac + Safari / Mac + Firefox / Linux + Firefox / Linux + Chrome

Hi @alexneo2003 , I'm hoping you can point me in the right direction. I'm using your Playwright Azure Reporter and I think i have everything configured correctly. Per your setup docs, I'm running many PlayWright tests written in TypeScript from ADO pipelines. The test results get posted to the "Test Run" tab on "Test Plans", works great! The #of tests pass or failing, get posted to the Pipeline run, works great! What I do not see is any data for the individual Test Case ID run, right click on test case “View Execution History”.

My question is, am I supposed too?

I'd love to see a working version of this.

Thank you in advance.
Randy

We use test.fixme() intentionally skip broken tests, test.skip() to skip if it is not applicable for other reasons.
Both gets reported to azure test plans as 'Paused', it would be nice if the test.fixme() would be reported differently.
Also a skipped due to previouse error in cobination with test.describe.configure({ mode: 'serial' }) is reported as 'Paused'.

The fixme / skipped / other state could be detected when the testCase.status is skipped and the testCase.annotations contain the fixme annotation / skip annotation / none of them.
The test.fixme() could then be reported as 'Paused' state, the test.skip() as 'Not Applicable' state and the others as 'Blocked' state.
(All states should exisits in azure testplans, as far is i can tell by using google)

The 'Blocked' could e.g. be reached when using test.describe.configure({ mode: 'serial' }) and the first test failed, than the following tests are 'Blocked'.

For the sake of backwards compatibility the config could provide the option the map theses state differently e.g.

{
   'fixme': 'Paused',
   'skip': 'Not Applicable',
   'other': 'Blocked'
}

As mentioned in #4 there are steps inside the azure testcases. It would be greate to connect the playwright test.step with the azure steps. To link these the same syntax like linking the testcases could be used, e.g. test.step("[2] login to the system") -> is linked to the second step.

This is a minimal example of a playwright test using steps:

test.describe('google div', () => {
  test('[31] failing is enabled', async ({ page }) => {
    await test.step("[1] Just a step", async () => {
      await expect(true).toBe(true);
    });
    
    await test.step("[2] Just another step", async () => {
      await expect(page.locator("div")).toBeEnabled();
    });
  })
})

The result of the steps are reflected correctly to the HTML report, so it should be possible to get the status of a step.
This might be done using the TestStep.error field, this should be investigated.

This may be more of a Playwright issue, but I thought I would ask here to see if anyone else had it as well:

I've been running playwright on ADO, and sending test result pass/fail back to ADO test cases with the playwright-azure-reporter
It is all working fine, and storing the screenshots, videos and trace files on the ADO test result. I set it to save this stuff in the config file:

attachmentsType: ['screenshot', 'video', 'trace'],

However, when I try to download a trace zip (from the ADO test case that has the execution result) and open it in the trace viewer (npx playwright show-trace), the viewer says it is not a valid trace file!

I've unzipped the file and tried to compare it against a trace zip produced locally on my computer - I don't see any difference between the two. The resources folder on the failing one has css, json, jpeg, etc files just like the first one does.

I get this error after running npx playwright test

(node:16704) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'value' of null
    at C:\Users\royal\source\repos\Playwright\node_modules\azure-devops-node-api\VsoClient.js:124:52
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:16704) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:16704) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
azure: Timeout while waiting for runId. Reporting is disabled.

part of my playwright.config.ts

import type { PlaywrightTestConfig } from "@playwright/test";
import { devices } from "@playwright/test";
import { AzureReporterOptions } from './playwright-azure-reporter'

const config: PlaywrightTestConfig = {
  testDir: "./tests",
  /* Maximum time one test can run for. */
  timeout: 50 * 1000,
  expect: {
    timeout: 30000,
    toHaveScreenshot: {
      threshold: 0.2,
      maxDiffPixels: 150,
    }
  },
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  //reporter: 'html',
  reporter: [
    ['html'],
    [
      '@alex_neo/playwright-azure-reporter',
      {
        orgUrl: 'https://myCompany.visualstudio.com/myCompany',
        token: 'les5gavgkxobxgsjfsdfsfsdfsdfsayksv2pw2jvzsc3hxuqy5unadq',
        planId: 138086,
        projectName: "myCompany",
        environment: 'myCompany',
        testRunTitle: 'Playwright Test Run',
        uploadAttachments: true,
        attachmentsType: ['screenshot', 'video', 'trace'],
      } as AzureReporterOptions
    ]
  ],
}

I copied the playwright-azure-reporter.ts from this repo directly to my root without any changes.
Did I miss any part in configuration?

Hi, thank you for a great package!

When enabling playwright-azure-reporter in playwright.config.ts, the normal console output seems to be hidden/overwritten by the reporter.

Without playwright-azure-reporter configured:

With playwright-azure-reporter configured:

Is there a way to disable this behavior? Setting logging: true in the config didnt do it.

when i want to run i get an error:

azure: Failed to create test run. Reporting is disabled.
undefined:1
Request timeout: /avanderonline/_apis/Location
^

SyntaxError: Unexpected token 'R', "Request ti"... is not valid JSON
at JSON.parse ()
at AzureDevOpsReporter.onBegin (C:\git\hydro-test\node_modules@alex_neo\playwright-azure-reporter\src\playwright-azure-reporter.ts:290:34)
at processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v20.11.1

Thanks Alex for the library it is really useful. Would you know if it is possible to configure the pipeline so we would leverage System.AccessToken and not a PAT? Otherwise maybe a service connection?

This is the error that i get and i cannot figure it out.

Failed to create test run. Reporting is disabled.
undefined:1
Cannot read properties of null (reading 'value')
^

SyntaxError: Unexpected token 'C', "Cannot rea"... is not valid JSON
at JSON.parse ()
at AzureDevOpsReporter.onBegin

We are currently running different browsers sequentially in different stages to inject the correct configurationIds to the testRunConfig. So the TestRun in the Azure UI is mapped to the correct configuration.

In the future we would like to run the tests in one stage to with multiple workers to reduce the runtime of the tests.
This will allow us to implement a locking mechanism to ensure the tests with different browsers will not disturb each other.

To make this working we need the option to set multiple configurationIds inside the testRunConfig or a way to create the testRunConfig dynamically. To create the testRunConfig dynamically would be a small change in the plugin, enabling alot of other usecases too.

One way to achieve this would be to have a testRunConfigCallback inside the plugin configuration.
The callback could be called in _createRun with one argument, the TestCase and use the TestCase.parent().project() to decide which testRunConfig to return.

been trying to fix this issue for while but I cant figure it out so reaching out for help!
set up playwright-azure-reporter in my project and I am able to report to the test cases when I run them on my laptop (locally). but when the tests are being run on azure pipeline, I'm getting the following error! do you know what the issue might be?
azure: Failed to create test run. Reporting is disabled.

Not sure if it is a bug, but if a test is associated with an unexisting ID, even if a comprehensive error message is displayed, it cancels the entire publishing and ignore all valid tests, rather than just skipping the invalid one and publish.

azure: Using run 51 to publish test results
azure: Start publishing test results for 44 test(s)
azure: Invalid test configuration!
azure: No test points found for test case [101] associated with test plan 31. Check, maybe testPlanId, what you specified, is incorrect.
azure: Test results published for 0 test(s)
azure: Run 51 - Completed

not able to find the attachment, which folder is it looking ?

Hi,

I'm not sure how to debug this but it's no longer possible to import the reporter module with 1.5.5:

If I downgrade to 1.5.3 (1.5.4 was not published to npm) it works again.
I'm using VSCode and TypeScript 5.0.4.

Hello,

Could you add a section on which rights are required for the Azure token please ? I noticed that many users just creates a full access token, which is not very secure.

Thanks ;)

@alexneo2003 Seems like there is a bug where if the test case once fail / blocked n in next run its passes , the status of the test case remains fail / block though in actual it is pass

steps to reproduce :

1. Execute the test which fail in execution.
2. Test status in ADO is fail ( thats fine )
3. Fix the test case and run it again , this time test is passing

Expected Result :- the failed test case status should change from fail to pass

Actual result : Remains Fail

Hi. I just tried the packages and its awesome. I thought about using the DevOps API directly to update my test cases, but with this package it works out of the box. Thanks.

It is possible to set the automation status (status/storage/etc.) as well? I would like to associate my test cases. You already pick the test cases and set the outcome...the association should be not a big deal?

Hi I'm trying your reporter, it looks very very promising but I'm getting this error.

URLs and token are correctly configured. Any suggestion? Thank you!

azure: {"$id":"1","innerException":null,"message":"A potentially dangerous Request.Path value was detected from the client (:).","typeName":"System.Web.HttpException, System.Web","typeKey":"HttpException","errorCode":0,"eventId":0}
azure: Failed to create test run. Reporting is disabled.

Encountering the following error on initial setup.

"TypeError: Cannot read property 'orgUrl' of undefined"

azure.config.json

{
    "token": "token",
    "orgUrl": "https://myURL.visualstudio.com/",
    "projectName": "MyProject",
    "testPlanId": 1555
}

playwright.config.ts

import type { PlaywrightTestConfig } from '@playwright/test';
import { devices } from '@playwright/test';
import azureConfig from './azure.config.json';

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
// require('dotenv').config();

/**
 * See https://playwright.dev/docs/test-configuration.
 */
const config: PlaywrightTestConfig = {
  testDir: './tests',
  /* Maximum time one test can run for. */
  timeout: 30 * 1000,
  expect: {
    /**
     * Maximum time expect() should wait for the condition to be met.
     * For example in `await expect(locator).toHaveText();`
     */
    timeout: 5000
  },
  /* Run tests in files in parallel */
  fullyParallel: true,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: [
    ['list'],
    [
      '@alex_neo/playwright-azure-reporter',
      {
        token: azureConfig.token,
        orgUrl: azureConfig.orgUrl,
        projectName: azureConfig.projectName,
        planId:  azureConfig.testPlanId,
        environment: 'QA',
        testRunTitle: 'Playwright Test Run',
        uploadAttachments: true,
        isDisabled: false,
      }
    ]
  ],

@alexneo2003
I am not getting the property in testRunConfig to set the status of the test cases as automated , the doc which u have attached in the description of testRunconfig has the property listed ,
Could you plz help me how can i set it with an example

thankyou so much for your work ! Cheers

Thanks for this great library!

The exception handling when the testApi fails assumes that the caught error message is encoded using JSON. You should probably decode the message depending on the Content-Type response header. In my case I did not have access to ADO which results in a non-JSON exception:

> npx playwright test --headed
azure: [1] has title - passed
azure: get started link - passed
azure: Failed to create test run. Reporting is disabled.
undefined:1
connect ETIMEDOUT 13.107.42.20:443
^

SyntaxError: Unexpected token c in JSON at position 0
    at JSON.parse (<anonymous>)
    at AzureDevOpsReporter.onBegin (C:\Users\SEPACOA\workspace\playwright.ts\node_modules\@alex_neo\playwright-azure-reporter\src\playwright-azure-reporter.ts:265:34)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v18.17.1

Additionally, would you be able to provide this library (and instructions) for JavaScript projects too?

I'm trying to get the testPointMapper to work when we run two browsers at once that are separated by Project. I have multiple configurations assigned to a test case in my Azure Test Plan. If I manually set the configurationIds to use both of the configurations it will find both of them and upload the results to both configurations.

As soon as I include the testPointMapper logic I get the following error:

I'm sure I'm just configuring the testPointMapper and configurationIds incorrectly in my config but I don't understand what's wrong. Thanks!

    [
      '@alex_neo/playwright-azure-reporter',
      {
        orgUrl: MY_ORG,
        token: process.env.AZURE_TOKEN,
        planId: MY_PLANID,
        projectName: MY_PROJECTNAME,
        environment: MY_ENV,
        logging: false,
        testRunTitle: MY_TESTRUNTITLE,
        publishTestResultsMode: 'testRun',
        uploadAttachments: true,
        attachmentsType: ['trace'],
        testPointMapper: async (testCase: TestCase, testPoints: TestPoint[]) => {
          switch(testCase.parent.project()?.use.browserName) {
            case 'chromium':
              return testPoints.filter((testPoint) => testPoint.configuration.id === '65');
            case 'firefox':
              return testPoints.filter((testPoint) => testPoint.configuration.id === '66');
            default:
              throw new Error("Invalid test configuration!");
          }
        },
        testRunConfig: {
          owner: {
            displayName: MY_NAME',
          },
          comment: MY_COMMENT,
          configurationIds: [65,66],
        },
      } as AzureReporterOptions,
    ],

Hi,
First of all, thanks for creating this project. it works great

I'd like to request the ability to associate azure test cases with playwright automated tests. at a glance this might look like out of scope of the project, but this is extremely useful to gather automation test coverage for UI Tests which provides value in the team, so would appreciate if you can consider it

the process consists in calling an Azure API to create or update an association https://dev.azure.com/{organization}/{project}/_apis/wit/workitems/{id}?api-version=7.0 (more info https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/work-items/update?view=azure-devops-rest-7.0&tabs=HTTP)

request body example: this changes manual to automated to a test case with the following values:
[
{
"op": "add", ("add" to new association "replace" to update existing association)
"path": "/fields/Microsoft.VSTS.TCM.AutomatedTestName",
"value": "CreateUser"
},
{
"op": "add", ("add" to new association "replace" to update existing association)
"path": "/fields/Microsoft.VSTS.TCM.AutomatedTestStorage",
"value": "FrontEndAppName"
},
{ //ignore this in case of updating existing association
"op": "add",
"path": "/fields/Microsoft.VSTS.TCM.AutomatedTestId",
"value": "NewGuid"
}
]

Attached is an example of how would look like:

Hi, I want to reference pipeline in the test report. However my pipeline is in different project than the Test Plans.
Is there a possibility to do a cross reference?

BTW, thanks for this project it helped me a lot.

Hi. I'm trying to configure this reporter and I can't see results in build view. I have created test plan having one passing and one failure tests. The results are visible in the test plan

but in the build view hasn't the results

Do I need additionaly configure "PublishTestResults" task like here?

Should I use this reporter? When I'm using JUnit reporter the build view has the results, but there are not snapshots attachments.

Can you explain me, how to use this reporter to see test results with attachments in build view?

Thank you in advance.

By any changes can we pass multiple plan ids in the configuration?? Since each test plan have different plan id

thanks !

@alexneo2003 I can make sure that there is something wrong and requesting you to have a look .

i have tried many times and able to reproduce this issue.

Note :- All the test cases Ids are different
Either its testRun / testResults , I am able to reproduce in both the scenarios

Seems like there is a bug where if the test case once fail / blocked n in next run its passes , the status of the test case remains fail / block though in actual it is pass

steps to reproduce :

Execute the test which fail in execution.
Test status in ADO is fail ( thats fine )
Fix the test case and run it again , this time test is passing
Expected Result :- the failed test case status should change from fail to pass

Actual result : Remains Fail

It would be nice if you could group different tests and have the common test case id on the test.describe('[123] Test name', () => {})

I have multiple suites in my test plan. Therefore, a test case can exist in multiple suites. For example, there are suites per stage (DEV/TEST/PROD). Only tests from a specified suite are executed in the build pipeline. The Playwright Azure Reporter is now publishing the results to all test cases. But I only want them to be published in a specified SuiteId. I didn't find the option in testRun.configurationIds or/and testPointMapper. I also don't want to create configurations.

Is there are way to add a SuiteId?

as part of the update of the test case in Azure DevOps, can we have the Test step details also copied in to test run as well as associated the attachments with each step(see attached screen grab)?

There is a cryptojs dependency which seems to be unused and seems to be not maintained anymore.
The last push to their repo was 11 years ago, i don't think there is any support for this.

Background is, i am currently checking if we will use this plugin in our pipeline and i have to ensure all the components are maintained and have a valid license.

We have test cases that use different data and likewise are hosted in separate Test Plans in ADO.

Suggesting the planId be changed to accept a string, then convert to a number within the code where necessary.

This would allow for the use of environment variables in the pipeline to control which test plan gets updated.

Hi, first of all thanks for this reporter, it's super useful!

I'm very new to Azure DevOps and noticed it's possible to set configuration options for test plans:

Is it possible to apply/create them dynamically and assign them to the executed test cases?
Right now, it always sets the default configuration for each test case:

I run the tests in three browsers and it's hard to see which browser was used for each test right now. It would be great to see the browser name in the configuration column there.

Hi, thank you for the great package, helps me a lot.

I was trying to get my test results posted but i was getting this error. The error suggests that the issue is between the test case and the test plan.

However, the actual issue was that i was passing a test configuration in AzureReporterOptions that was not properly assigned to the test case in azure. I was thinking the error message could be improved? Cheers :)

https://dev.azure.com/XXX/XXX/_testPlans -> Right click on a test plan/case -> Assign configuration

azure: Start publishing test results for 1 test(s)
azure: No test points found for test case [107222] associated with test plan 75767. Check, maybe testPlanId, what you specified, is incorrect.
azure: Test results published for 0 test(s)
azure: Run 1736182 - Completed

To be clear, the error message is correct. My configuration was wrong. But the fact that it did not mention the test configurations kind of through me on the wrong path.

Would it be possible to make use of custom annotations for test plan ID linking ? That way we could avoid polluting test titles ;)

Example :

test('my test title', async ({ page }) => {
  test.info().annotations.push({
    type: 'azureTestPlanID',
    description: '42',
  });
  // ...
});

This is indeed more verbose, but adding custom decorator support would fix this. The same approach can be seen in the playwright-bdd package, and it is very handy !

Example :

@azureTestPlanID(42)
test('my test title', async ({ page }) => {
  // ...
});

What are your thought about this ?

Test publishing by 50 counts, but logging decrementing by 5

Hi @alexneo2003 ,

Some of my tests in ADO are having dynamic parameters and I need to run them in iterations based on the number of records in the parameters table.
How can I handle such a situation with your implementation?

https://learn.microsoft.com/en-us/rest/api/azure/devops/test/iterations/get?view=azure-devops-rest-7.1&tabs=HTTP

Thanks!

Hi @alexneo2003,

Can we have a feature switch ON/OFF in playwright to enable/disable (with the enabled option as default) the reporting of test runs in Azure DevOps?

Something like:

**enableAzureReporter: 'Yes'**

`[
      '@alex_neo/playwright-azure-reporter',
      {
        enableAzureReporter: 'Yes',
        orgUrl: 'https://dev.azure.com/your-organization-name',
        token: 'AZURE-TOKEN',
        planId: 44,
        projectName: 'SampleSample',
        environment: 'AQA',
        logging: true,
        testRunTitle: 'Playwright Test Run',
        publishTestResultsMode: 'testRun',
        uploadAttachments: true,
        attachmentsType: ['screenshot', 'video', 'trace'],
        testRunConfig: {
          owner: {
            displayName: 'Alex Neo',
          },
          comment: 'Playwright Test Run',
        },
      } as AzureReporterOptions,
    ]`

Thanks!