jdk20 MiniMix bigdecimal about openj9-systemtest HOT 10 CLOSED

related: adoptium/aqa-systemtest#482

from openj9-systemtest.

@llxia @Mesbah-Alam pls take a look.

from openj9-systemtest.

Hi @pshipton ,

net.adoptopenjdk.test.bigdecimal.TestSuite057 is not excluded in JDK 19, it's actually working normally on 19 : https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_aarch64_mac_Nightly_testList_1/117/consoleFull.

org.junit.ComparisonFailure: expected:<-9[E+18]> but was:<-9[000000000000000000]>

Is this failure on JDK20 aarch64 Linux indicating a real issue?

Similar failure is not seen on JDK20 AIX : https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_ppc64_aix_Nightly/1/tapResults/.

from openj9-systemtest.

If you look at the build https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK20/5/ you can see that different platforms are failing different tests. Can you pls try the tests on Adoptium and see if they similarly fail. OpenJ9 didn't change any related behavior for jdk20 afaik.

from openj9-systemtest.

I tried some grinders on the latest and the failures are still occurring with OpenJ9.
https://openj9-jenkins.osuosl.org/job/Grinder/2087/
https://openj9-jenkins.osuosl.org/job/Grinder/2088/

from openj9-systemtest.

Running on an Adoptium jdk20 build also fails.
https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/2089/

17:06:05  openjdk version "20-beta" 2023-03-21
17:06:05  OpenJDK Runtime Environment Temurin-20+36-202303170331 (build 20-beta+36-202303170331)
17:06:05  OpenJDK 64-Bit Server VM Temurin-20+36-202303170331 (build 20-beta+36-202303170331, mixed mode, sharing)

from openj9-systemtest.

From the readme: "The Math suite contains simple tests which are designed to exercise the Math apis".

Are these math APIs supposed to generate different results on different platforms?

Looking at a failed test method on AIX in https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_ppc64_aix_Nightly/1/tapResults/ :

 testItem_0328(net.adoptopenjdk.test.bigdecimal.TestSuite033): expected:<6[E+2]> but was:<6[00]>

The same test passes on Aarch64 Linux in Adoptiom JDK20 :

17:09:57  LT  21:09:56.909 -   33 JUnit[net.adoptopenjdk.test.bigdecimal.TestSuite033]  Weighting=1 

Similarly, it passes on Aarch64 Linux on OpenJ9 : https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_linux_Nightly/1/tapResults/.

from openj9-systemtest.

Even on the same platform, different tests fail between OpenJ9 and Adoptium, and different tests fail from run to run. There is something else going on. Perhaps a random sub-set of tests is being run instead of a consistent set.

I looked at all of the particular failures below, and they all have checks for JavaSpecVersionChecker.isJDK13OrNewer() but the behavior has reverted to the pre Java 13 behavior. I'm guessing if you update isJDK13OrNewer() to return false for jdk20 then the tests will pass.

OpenJ9:
https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_linux_Nightly/1/tapResults/
https://openj9-jenkins.osuosl.org/job/Grinder/2088/tapResults/
Adoptium:
https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/2089/tapResults/

from openj9-systemtest.

I'm guessing if you update isJDK13OrNewer() to return false for jdk20 then the tests will pass.

Yes, confirming that that is indeed the case.

Also, following adoptium/aqa-systemtest#482, updated the test to check for 12 < version < 20, instead of version > 12 and it passes in jdk20. Grinder links in following PR : adoptium/aqa-systemtest#483.

from openj9-systemtest.

The MiniMix_5m test is now passing on JDK 20 https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_mac_Nightly/8/tapResults/.

Can this issue be closed?

from openj9-systemtest.