Need for enhanced visibility into test flakiness directly within Allure reports about allure2 HOT 13 OPEN

@ShivaPotlapelli, I have no relation to the Allure Framework team. You must have confused me with someone else. 🤔

from allure2.

Wait, but there is a "Flaky tests" filter there on UI:

I understand there is an issue with displaying the count of tests, but does this filter work for you, in the first place?

from allure2.

@noomorph I seen that.. I don't want the flaky of a test result from the run..I have an 3rd party tool known as reportportal.io
which will give me from the last 5 runs this tests are flaky and those tests we don't run in the suite but we just wanted to show the number this many tests are not run and an hyper link redirecting to the third party containing those flaky tests

from allure2.

@ShivaPotlapelli In any case, this doesn't look like the correct repository for this issue.

The brains of the reporter and Backbone.js UI are here: https://github.com/allure-framework/allure2

from allure2.

@noomorph can u migrate this to there in that case..i don't see the repository that u redirected has any specific place to log this..it asks if there integration issues with tech stack . since am using playwright-allurejs ..i logged this issues here.

from allure2.

@noomorph is right, it doesn't relate to javascript integration. I'll transfer the issue for you.

from allure2.

@epszaw can you please let us know if there is any thing u could suggest on the above request made

from allure2.

@ShivaPotlapelli honestly, I think this thing could have been implemented as a plugin.

from allure2.

Is there a way to customise the report and show it ...I tried via env variables but that not the place where i wanted my flaky number to be appeared..Is there any other way i could do it??

from allure2.

It is possible to create a Backbone/Marionette.js plugin. The plugins are not well documented but there are some bits to get started: https://web.archive.org/web/20230321171430/https://docs.qameta.io/allure/#_allure_plugins_system

from allure2.

@noomorph can someone from u team start on the above

from allure2.

@baev can u look into above

from allure2.

@ShivaPotlapelli, I appreciate your request.

Currently, Allure Report provides flaky marks for tests based on test history (and allows users to provide it manually from tests using RuntimeAPI).

If I understand it correctly, you are asking for:

1. Reimplement widgets and graphs in the report to include information about flaky tests in there
2. Allow using 3rd party tools to supply information about flaky tests

As @noomorph mentioned correctly, all of the above can be implemented using the Allure plugin API.

However, each item is challenging to implement in a general way (so we can push the changes to all the Allure report users). Some of the problems to think of:

1. How do you link the report portal test case to the Allure test result?
2. How do we include an additional flaky category to existing widgets? Many Allure report users rely on the results counts, but since tests can be flaky and broken/failed at the same time, we need to think of a solution.
3. For large reports, there is a chance to ddos 3rd party tool while trying to access the data

So, having all that in mind, what can we do:

1. Work through the request, splitting it into smaller tasks and prioritising them separately. We have a lot of requests regarding flaky test reporting already, and we value your contribution as well.
2. If you urgently need to improve the test reporting — feel free to try Allure TestOps. It may suit your needs already
3. There is a possibility to buy paid support from our team as well (works best with Allure TestOps — we have a dedicated solution architects team for that)

from allure2.