Publish test results only for specific test suites about playwright-azure-reporter HOT 9 CLOSED

Your reporter already has the option I need, to publish test results only to a suite. After extensive research and consideration of implementing our own reporter, I came across Azure "TestPoints" (Testpoints), and it can be realized through the testpointMapper.

Example:
testPointMapper: async (testCase: TestCase, testPoints: TestPoint[]) => testPoints.filter((testPoint) => testPoint.suite?.id === 'suiteId'), // suiteId = id of the suite in devops

The ticket can be closed.

from playwright-azure-reporter.

how you handling running only specified suite?

I read all tests from a specified/given azure devops suite by using the devops api and after that I compare that result with the ids from the playwright tests (and run only the matching tests).

Your suggestion does not work for me, because I dont want to add the suiteIds to the Playwright tests. The suites can be changed or new ones added, so that its a big effort to manage suites in the playwright code. In my build pipeline configuration I already pass the planId and suiteId...

You already publish the test results by using a devops testplan id (planid in your configuration). From my understanding it should be also possible to add an additional value to the reporter config for a suiteid so that the results can be published only to that devops suite. So as an optional config parameter "suiteId"...

'@alex_neo/playwright-azure-reporter',
      {
        orgUrl: 'https://dev.azure.com/your-organization-name',
        token: process.env.AZURE_TOKEN,
        planId: 44,
        suiteId: 62, // [62,63,64]
        ....

from playwright-azure-reporter.

We can use it, yes, but he problem is that the current implementation of the reporter publish the results to all suites containing the same test case.

Imagine you have :

• Testplan
  • Suite A
    • Testcase 1
    • Testcase 2
  • Suite B
    • Testcase 1
    • Testcase 2
    • Testcase 3
  • Suite C
    • Testcase 1
    • Testcase 4

Now, without any other config the reporter publish the result for Testcase 1 + 2 to all 3 devops suites. But I only want to have the results published in "Suite A" because I ran this suite. If you imagine that the 3 suites are running on different environments (Suite A = DEV environment, Suite B = TEST environment, ...), it distorts the result. In suite A (DEV) the test failed, suite B (TEST) the test passed. Now, in suite B, it is also marked as failed, but its still passed.

Publishing test results has to be thought of on different levels; not everything ends up in one pot. Like you implemented it with the configurations - if I understand this configuration correctly. The results are also published for each configuration (Chrome, Firefox, etc.). I already thought about using your reporter with the configurations (testPointMapper) but it would be easier to add only a suiteId.

from playwright-azure-reporter.

You can add your SuiteId to describe title, like

test.describe('(675645) root test suite', () =>

And than use it with --grep, like

npx playwright test --grep 675645

from playwright-azure-reporter.

Thanks for the quick reply.
I meant the SuiteId from devops, not from the playwright test code (or adding it to the test code).

In the playwright.config you can currently only specify the azure planId and not a subordinate suite. If I only add a planId, all test cases in all suites will be updated with the outcome (in my case, the same test case can exist in different devops suites). So what I need is to specify the suiteId in the reporter (devops planId + suiteId).

from playwright-azure-reporter.

@taddeuz yep, I understand what you mean

you wrote

Only tests from a specified suite are executed in the build pipeline.

how you handling running only specified suite?

If you specify suiteId inside reporter options you still need to handle with what you need to run from code!
otherwise you will run ALL tests but publish only for specified suiteId? yes?

I suggest you:

1. to supplement your describe title with specific suiteId which related to you test plan
2. you can define more than one suiteId inside describe title, like

 test.describe('(2737823,34590,2382313) Login Page @UI', () => {
	test('[2343] Test one', async ({ page }) => {})
	test('[2344] Test two', async ({ page }) => {})
	test('[2345] Test three', async ({ page }) => {})
})

3. define project inside playwright.config.ts file and add the grep parameter for each, like

{
	name: 'web-dev',
	testDir: 'specs/web',
	grep: new RegExp(/2737823|34590/)
}

4. define run script inside package.json to run test only for specific project

"scripts": {
  "test:web:dev": "playwright test --project=web-dev"
}

when you run script "test:web:dev" pw will run only specified suites and reporter publish results only for runned tests

from playwright-azure-reporter.

@taddeuz
ok, you want:

1. specify suiteId like one or array
2. run tests
3. then reporter should make some request to get all testCaseIds related to this suiteId
4. somehow process test results

we need to decided how playwright will run only related testCases? otherwise playwright have to run ALL tests at startup.

for playwright you need define tests at startup otherwise he runs all tests

or if you want handle with tests through reporter, like in onTestBegin hook and check whether this test is included in the selection of tests that correspond to specified suiteId. and if not - skip test and not publish it result through reporter

you wrote

You already publish the test results by using a devops testplan id (planid in your configuration). From my understanding it should be also possible to add an additional value to the reporter config for a suiteid so that the results can be published only to that devops suite.

it's not a matter of what is published, but of the fact that you need to decide how to launch only what you need
because otherwise ALL tests will be launched, but only the necessary ones will be published - this is redundant

from playwright-azure-reporter.

1. specify suiteId like one or array
2. run tests
3. then reporter should make some request to get all testCaseIds related to this suiteId
4. somehow process test results

Thats what we want.

Just for your information how we implemented a workaround/solution to only execute tests within a azure devops suite:

1. Get all tests from a test suite from azure devops (getTestCases() call, using the azure-devops-node-api)
2. Extract the test case ids from that call --> now we have all test cases which are added to a azure testplan suite
3. Lookup into our Playwright tests if there are is a matching id and if yes, push that into an array
4. Execute all the matching tests (with grep)
5. Now we are able to specifiy a suite id when we start a azure test build and we can decide, which devops suite should be executed. This gives us maximum flexibility because we can now put together a suite of test cases in Devops without having to make any changes to the code.

That implementation is not really related to the playwright-azure-reporter. Now we want to use your package to publish the results (or we have to implement an own reporter).

from playwright-azure-reporter.

OK, if you want use this report to publish results - use it. You no need any features with suiteId. Report will report results only for runned tests.

Your feature with suiteId will not work in reporter as well because reporter will run after you start tests. In order to run tests with grep reporter should be run before but it is not so.

from playwright-azure-reporter.