[Regression]: Artifacts created during test execution get random string in the name about playwright HOT 4 CLOSED

where we compare actual files with the current existing ones.

In which directories are you comparing them? We did a change but this should only have affected the test-results directory, not the actual golden test expectations on disk. I tried to reproduce it on my machine, and there it was working as expectd (only test-results have changed).

// 1.42 /Users/maxschmitt/Developer/tmp/1510/test-results/project1-Here-is-a-test-with-a-very-long-title-5bd13-long-title-tlong-title-to-reproduce-the-problem-test
// 1.43 /Users/maxschmitt/Developer/tmp/1510/test-results/project1-Here-is-a-test-wi-5bd13-le-to-reproduce-the-problem-test

Do you consume them via test.info().outputDir or via the reporting api?

from playwright.

Yes, we use the files outputted in the test-results folder. We have an manual approval step on failed test runs on CI, that if approved, runs a script that compare file names and update the golden snapshots using the actual ones from test-results.

Can you please confirm that this change is only affecting file names?

from playwright.

It only affects the test-results directory which is not intended to be consumed directly. Its intended to be consumed via our APIs, either inside a test via test.info().outputDir or via the reporter API.

from playwright.

Yes, we use the files outputted in the test-results folder. We have an manual approval step on failed test runs on CI, that if approved, runs a script that compare file names and update the golden snapshots using the actual ones from test-results.

As Max mentioned you can implement a custom reporter and access all the images including their paths via TestResult.attachments. The exact naming of the paths in toHaveScreenshot is an implementation detail and it can change between versions.

Can you please confirm that this change is only affecting file names?

The change affected file names of the diff and actual images. Actual snapshot name stays the same. It can

from playwright.