Lighthouse fails to produce accessibility score altogether for incomplete axe-core result about lighthouse HOT 4 OPEN

Thanks for filing a bug with axe-core, we will close this once it's fixed and the dependency is updated.

An accessibility score to still be calculated to prevent breaking PR gates completely based on score and fail to see general progress

If an axe check throws an error, we don't know if the audit passes and so we can't know what the accessibility score would be.

Maybe we can treat the audit as failing which could be less punishing than having no score at all, but I'm not completely sold on the idea.

A link / more resources to report the incomplete result to axe-core

I think this could be useful. 90% of the time issues related to accessibility we wind up upstreaming to axe-core.

from lighthouse.

@rajsite It was fun to walk back the history in ni/nimble#1650 and see everything. In this case I think you managed to find a really nice bug in axe. It's wildly infrequent we see those sorts of DOM level exceptions from axe. So.. hats off to you for really stressing the system. :)

from lighthouse.

Maybe we can treat the audit as failing which could be less punishing than having no score at all, but I'm not completely sold on the idea.

In our case we had to disable the CI gate for the accessibility check for several weeks (until we had time to come up with a reproducing example and make the reports) in which time other accessibility issues that would have been caught otherwise were allowed through.

from lighthouse.

We are not going to change how the accessibility category is scored since there is value in displaying errors loudly. Reporting this error to us when it happens is exactly what we want :)

If you weren't doing this already, you can use skipAudits to disable a single erroneous audit without disabling the entire accessibility score check.

I will leave this issue open because there is still an upstream issue and we should update the accessibility category description and docs to mention Axe and direct users to file bugs there.

from lighthouse.