[Reporting bug] Tests that passed on retry are not being reported accurately about bluepill HOT 4 OPEN

I saw similar behavior, and the root cause was a small bug in the collation of test results: #515

Are you seeing invalid XML files created in any of your executions? If an execution produces a Junit XML file that isn't a valid XML document (which we were seeing when a simulator failed to boot; the file contained just the string "NO LOG: JUnitReporter"), other (perfectly valid) test reports would be treated as invalid, and not merged into the final combined report.

from bluepill.

True. I see the executions where this did happen and I see how it could skip the subsequent reports that might be perfectly valid. Thanks for the fix.

from bluepill.

@jschear Your PR(#515) might fix this issue. I will resolve this issue after verifying.

from bluepill.

This doesn't seem to be fixed. I'm using 5.11, and I see results for each individual attempt in the xml, so if a test fails, retries, and then passes, I still have a failure in the output. There's an old open PR to add a flag for this here: #467

...and one comment even mentions making it the default behavior, but that doesn't seem to have happened.

from bluepill.