[Feature]: Dynamic testRunConfig about playwright-azure-reporter HOT 10 CLOSED

@alexneo2003 thanks for the offer to create a beta later on, i do not expect you to active check this or work on this.
The main reason for me to write updates here is to keep you / others updated there is still progress in this feature request.

from playwright-azure-reporter.

1. i'm not using configurationIds in my projects and have no expertise with him
2. if looking at ADO doc we can find

configurationIds | integer[] | IDs of the test configurations associated with the run.

I assume that you can pass several configurationIds and it should be attached to testRun, i think so, but i can't to check this

3. one tests running === one test run at ADO. as alternative for running your tests i think you can write something like this
   you need npm-run-all package
   and next running scripts

  "scripts": {
    "test:chromium": "npx playwright test --project=chromium",
    "test:firefox": "npx playwright test --project=firefox",
    "test:webkit": "npx playwright test --project=webkit",
    "test:all": "run-s -c test:chromium test:firefox test:webkit"
  },

the main thing - pass -c flag in run-s it will allow run next test project when last was failed
as result you receive: parallelism within each project, all of you projects will be running separately, ADO testRun for each project

from playwright-azure-reporter.

1. i'm not using configurationIds in my projects and have no expertise with him
2. if looking at ADO doc we can find

configurationIds | integer[] | IDs of the test configurations associated with the run.

I assume that you can pass several configurationIds and it should be attached to testRun, i think so, but i can't to check this

Yes, it is possible to use multiple configurationIds, but the testcase run will be linked in the UI only one configurationID.
(I am not sure about the logic, looks like the lowest ID is set, the order is ignored)

I don't know why there is an array in the API, but thats the observed behaviour of the UI.

3. one tests running === one test run at ADO. as alternative for running your tests i think you can write something like this
   you need npm-run-all package
   and next running scripts

  "scripts": {
    "test:chromium": "npx playwright test --project=chromium",
    "test:firefox": "npx playwright test --project=firefox",
    "test:webkit": "npx playwright test --project=webkit",
    "test:all": "run-s -c test:chromium test:firefox test:webkit"
  },

the main thing - pass -c flag in run-s it will allow run next test project when last was failed as result you receive: parallelism within each project, all of you projects will be running separately, ADO testRun for each project

However we achieve the parallelism or even run sequentially, there is the need to set the corresponding configurationIds for each run to get it mapped in the UI. So there is some logic required to control the configurationIds and a callback for the whole testRunConfig would be a flexible way for this.

from playwright-azure-reporter.

Yes, it is possible to use multiple configurationIds, but the testcase run will be linked in the UI only one configurationID. (I am not sure about the logic, looks like the lowest ID is set, the order is ignored)

I don't know why there is an array in the API, but thats the observed behaviour of the UI.

Oh, yes. ADO api has some stranger things)

However we achieve the parallelism or even run sequentially, there is the need to set the corresponding configurationIds for each run to get it mapped in the UI. So there is some logic required to control the configurationIds and a callback for the whole testRunConfig would be a flexible way for this.

you can use some environment variable to pass it into config
like

"scripts": {
  "test:chromium": "cross-env CONFIG_ID=15 npx playwright test --project=webkit"
}

playwrignt.config.ts

reporter: [
    ['list'],
    [
      '@alex_neo/playwright-azure-reporter',
      {
        orgUrl: 'https://dev.azure.com/your-organization-name',
        token: 'AZURE-TOKEN',
        planId: 44,
        projectName: 'SampleSample',
        environment: 'AQA',
        logging: true,
        testRunTitle: 'Playwright Test Run',
        publishTestResultsMode: 'testRun',
        uploadAttachments: true,
        attachmentsType: ['screenshot', 'video', 'trace'],
        testRunConfig: {
          owner: {
            displayName: 'Alex Neo',
          },
          comment: 'Playwright Test Run',
+         configurationIds: [process.env.CONFIG_ID || 1]
        },
      } as AzureReporterOptions,
    ],
  ],

from playwright-azure-reporter.

We are currently using the environment variable way: configurationIds: [Number(process.env.CONFIG_ID)],.
But this will block an local locking mechanism, to ensure some tests are not running concurrently.
Which was the reason to raise this ticket, to have dynamic way to set this.

Some more background to the locking mechanism, we need this to ensure some external interfaces with a limited working set of data are not called concurrently. Or later one when we integrate some hardware into the process to ensure the hardware is not used concurrently.
If we could use workers for concurency, we could just use a lock file (or whatever) to implement a local lock on the system.
As soon as we start to use different stages, etc. we are runnig on different VMs and need a external locking mechanism.

from playwright-azure-reporter.

I gave it a try, see this branch.

The Suite is passed to the Callback, it should be possible to get the project configuration from there.
I am not sure about saving the Suite in the instance, this might make some problems (in mode == testRun).

And all changes are currently not tested at all ...

from playwright-azure-reporter.

I had a closer look now and it looks my change is not usefull in our case.
We need a custom _getTestPointIdsByTCIds implementation to pick the correct test point of this test run.

from playwright-azure-reporter.

i'm appreciate for you work but i can't check this because i haven't opportunities. we don't using such scenarios.
if you can and if you have expertise in this area you can create PR with this features.
but you must be sure that it works as it should.
i will publish you PR as beta package for testing.
you are welcome

from playwright-azure-reporter.

@alexneo2003 i created the PR #18 with the change.

If there are any questions to the change, just add comments to it and i will try to answert them.

from playwright-azure-reporter.

@alexneo2003 i created the PR #18 with the change.

If there are any questions to the change, just add comments to it and i will try to answert them.

yes, i'm seen it.
i'm realy appreciate for you work.
I have a heavy workload at work and have no time to review you PR, i will try to do it soon

from playwright-azure-reporter.