Publish artifacts using JDK 1.8 or remove sbt JDK 1.8 check? about pekko HOT 14 CLOSED

👋 it was good to meet in person!

Likewise!

So best in the short run would be to build with JDK11, but have JDK8 available on the build machine as well. I think on most systems this should be auto-detected (and throw an error when it is not found, in which case you can point a JAVA_8_HOME environment variable to it).

So just to clarify what is going on, it appears that to properly build Pekko (and by build I mean JVM artifacts which are shipped to Maven repo) we need a machine which

1. Has a JDK 11 installation available which I presume can have the standard installation (i.e. JAVA_HOME) point to this. Necessary for docs and java.util.concurrent.
2. Also has a JDK 8 installation available which JAVA_8_HOME needs to point to. From reading the code the Java 8 installation is not used for actually compiling Pekko, the only reason we need it is to grab runtime rt.jar which we compile with JDK 11.

From a quick look at the code it does appear to automatically handle which artifacts need to compiled with which JDK. In any case the reason why I am asking this is that it needs to be documented as part of the release process (and also checked/improved/modified) because a proper ASF release needs to be done on a release managers machine, not CI.

It also makes sense that for nightlies/snapshots we also do the same behaviour, if I am correct about JDK 8 this shouldn't be too hard to solve (i.e. we can manually install JDK 8 to some path on a machine that already has JDK 11 and just point JAVA_8_HOME to it).

from pekko.

Answered in #130 (comment) since its a more relevant thread

from pekko.

I am closing this since all questions have been answered, i.e. we are going to release with JDK 8 support.

from pekko.

So I spoke with @raboof at FOSDEM, and the situation with pekko build is quite interesting and complicated. While building documentation only works with JDK 11 (as has been commented in the various github actions workflows), the core build/artifacts of Pekko needs to be built with JDK 1.8.

While typically one would use source/target tags (such as https://github.com/pjfanning/sbt-source-dist/blob/main/build.sbt#L9), specifically in the case of Pekko this doesn't work because while the source/target flags force javac to produce the relevant targeted bytecode, it doesn't account for JVM standard library changes and apparently there was some ABI incompatible standard library changes from JDK 1.8 to JDK 11.

from pekko.

👋 it was good to meet in person!

I would definitely recommend compiling with JDK11 or later, to unlock some things related to java.util.concurrent and JFR, even if for 'the rest' of the artifacts we support running on Java 8.

Indeed normally you'd just build with JDK11 and target Java 8, but because of some JVM standard library changes, for some classes (notably EnvelopeBuffer) this isn't sufficient, and you need to provide a reference to the JDK8 runtime library. Additionally, I think some of the sun.misc.Unsafe usage should still be removed before this is possible.

So best in the short run would be to build with JDK11, but have JDK8 available on the build machine as well. I think on most systems this should be auto-detected (and throw an error when it is not found, in which case you can point a JAVA_8_HOME environment variable to it).

More details at https://github.com/apache/incubator-pekko/blob/main/project/JdkOptions.scala#L60-L67 and https://github.com/apache/incubator-pekko/blob/main/project/JdkOptions.scala#L77-L79.

from pekko.

Yes, that looks accurate.

the reason why I am asking this is that it needs to be documented as part of the release process (and also checked/improved/modified) because a proper ASF release needs to be done on a release managers machine, not CI.

It might be interesting to know it might be OK to create ASF releases from CI when certain conditions are met, something like:

• project has reproducible builds for source and binary artifacts
• RM creates tag
• CI system builds RC, signs it with a CI specific key and deploys it to staging
• PMC members review the release which includes building and confirming that source and binary artifacts are as expected (identical to those built by CI)
• PMC members VOTE
• RM promotes RC if vote passes

As you can see this hinges on having reproducible builds for Pekko - which might be a good thing to have in any case, but premature to require for the release? I've done some work towards that in the past, so it might be close, but I'm not sure it's currently complete.

(more context on the members list at https://lists.apache.org/thread/dytplxdj7cy545oxcvgl2wjombj3gbmc - we might want to clarify https://www.apache.org/legal/release-policy.html#owned-controlled-hardware further at some point)

from pekko.

Indeed normally you'd just build with JDK11 and target Java 8, but because of some JVM standard library changes, for some classes (notably EnvelopeBuffer) this isn't sufficient, and you need to provide a reference to the JDK8 runtime library. Additionally, I think some of the sun.misc.Unsafe usage should still be removed before this is possible.

We should check if that is even still correct, that sounds like a bug in one version of JDK 11 that might have been fixed in the meantime...

from pekko.

We should check if that is even still correct, that sounds like a bug in one version of JDK 11 that might have been fixed in the meantime...

This would be nice to verify in the sense that if has been fixed then there is less of a need to create a Dockerfile so we can setup a proper build environment for a release manager

from pekko.

I just checked and at least the Unsafe usage would have to be fixed. Additionally, just verifying that one issue might not be enough to validate that there might be other incompatibilities with JDK 8, so I guess we should keep it like this for now...

from pekko.

Do we understand how this is currently working on GHA? We only install JDK 11 but JDK 8 is automatically available on the ubuntu-20.04 of GHA?

from pekko.

Not sure whether that makes it magically work, but yes:
https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#java
(4 JDKs as it even packs Graal it seems)

from pekko.

Not sure whether that makes it magically work, but yes: https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#java (4 JDKs as it even packs Graal it seems)

Thanks but the environment variable is different here than what pekkos sbt seems to be checking? From https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#java it says JAVA_HOME_8_X64 but the sbt build checks for JAVA_HOME_8?

from pekko.

If JAVA_8_HOME is not set, the build consults the sbt fullJavaHomes, which among other sources auto-detects Java from common distribution paths

from pekko.

If JAVA_8_HOME is not set, the build consults the sbt fullJavaHomes, which among other sources auto-detects Java from common distribution paths

Thanks.

The good news is that then the CI building pekko-core is then a non issue so snapshots shouldn't have problems related to this thread. If I understand correctly they also happen to offer equivalent docker images (see https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#cached-docker-images) which then makes the just a lot easier for a release manager.

from pekko.