What would be your recommendation to native compile a simple Java library and then call this native compiled library from another Java program? about graal HOT 19 CLOSED

@letmedrawit I'm sorry I don't have a better answer, but we have this very same problem and we solve it with JNI.

The JNI Invocation API has a sample which shows exactly what you are asking. The steps are simple:

1. Build a native shared library
2. Describe it to JNI
3. Invoke it via JNI in Java in a new code unit
4. Build the code unit as a native binary

As demonstrated in the documentation, this technique works.

from graal.

That's totally fine, no worries at all.

1. Build a native shared library

The premisse is wrong. I do NOT have any native shared library to build. I have a pure Java library I would like to distribute in binary format. There is no C/C++ code involved.

from graal.

Hi @letmedrawit,

Thank you for reaching out to us!
Have you had the chance to look at the following documentation? I believe it could be of use to you in the use case you shared.

from graal.

Thanks @selhagani for your reply. I did check your link, but not sure it applies for my use-case. In my use-case my HashMap is implemented in pure Java, without any native methods. What I would like to do is:

1. Compile with AOT my HashMap.java code
2. Now I have a .so library (somehow)
3. Load this library and instantiate the HashMap class in Java
4. Use the HashMap class in Java as it was loaded from a regular jar in the classpath (but it was instead loaded from a native shared library)

Is it possible somehow?

EDIT: Just had a crazy idea. Maybe a native method (implemented in C/C++) that just instantiates and returns the HashMap object to Java? Well, I will probably need an interface for my HashMap (Map.java) so that I can make calls to HashMap in my Java code after calling this native method that instantiates the HashMap and returns it to my Java code.

from graal.

@letmedrawit the docs @selhagani linked are still relevant, because when you build your "framework" in JVM bytecode to a native shared library, it ceases to be a Java library, and is now addressable by JNI.

There are a few ways to load native libraries and use them in Java. JNA, JNI, and FFM would all be options with varying degrees of support in GraalVM. Generally speaking these are the only options regardless of how a native library is authored as source -- in other words, the output of Native Image is, to regular Java, roughly the same as any other native library.

from graal.

@letmedrawit not sure if your idea is really feasible. In your example, your HashMap, if compiled into and loaded from a native shared library, will run in a so called "Isolate". That is basically an independent JVM instance with its own heap within your Java process.
For something like the GraalVM Compiler (i.e. "libgraal") it makes sense to run in its own JVM with its own heap to not interfere with the host application and its in-/output is just a byte array (i.e. bytecode as input, native code as output). For a HashMap like in your example, the overhead of running that in an Isolate is probably too high.

from graal.

your HashMap, if compiled into and loaded from a native shared library, will run in a so called "Isolate". That is basically an independent JVM instance with its own heap within your Java process

I think you might be wrong here. When you call a shared library (i.e. native code) from Java code, there is not a second JVM involved.

EDIT: Just had a crazy idea. Maybe a native method (implemented in C/C++) that just instantiates and returns the HashMap object to Java? Well, I will probably need an interface for my HashMap (Map.java) so that I can make calls to HashMap in my Java code after calling this native method that instantiates the HashMap and returns it to my Java code.

I think that's the only feasible solution. Unless someone can come with something out-of-the-box which I'm not seeing.

from graal.

@simonis Native Image shared libraries accept an isolate as input; it is up to the outer program to manage isolate state.

@letmedrawit @simonis is explaining isolates as a concept similar to a JVM-within-a-JVM. this isn't actually what it is, it's just a metaphor to explain the isolation involved. It is true that with a shared library you will need to manage isolate state.

HashMap and some other Java types are banned from use in certain contexts under optimizing Truffle runtimes, but otherwise maps should be usable from a shared library, so long as you properly setup the isolate state and hand it to your app. That's required anyway though.

The "out of the box" solution is JNI, which has been supported since very early Java (like 1.1 iirc). Some people prefer JNA, which offers some developer ergonomics like generation of Java classes from native code layouts. JNI is rather strict and can be hard to work with.

FFM is the Foreign Functions & Memory APIs, which are newer and are not yet stable or fully supported in GraalVM. FFM is referred to sometimes as "Panama." Panama is more flexible and powerful than JNI without a loss in performance and without the build complexity of, say, JNA. It also relaxes some of the clumsiness of JNI. But you can only do upcalls on Linux x86 with GraalVM as of the latest release.

There is no other way to load native code in a JVM except perhaps to extend the JDK source code by hand.

Layered Native Images -- an upcoming feature -- may add another tool to the toolbox, but it has not shipped yet.

from graal.

Also, of course--a regular JVM bytecode library distributed via Maven works fine with Native Image, since NI effectively compiles JVM bytecode to native programs, based on the class and module paths.

You can embed configurations for GraalVM in your library distributions. Several popular projects do this (Netty, Micronaut, JLine, etc). These configuration files can register reflective access, resources, and so on, which apply downstream when consumed in a native build. You embed them in the JAR.

In some ways this is better than distributing a native library, especially for a framework, because the JVM bytecode can be compiled and optimized together with consideration of the end user's code. Dead code elimination can also be more effective.

But if you really do want to distribute a shared native library, you can; it's just that your options are the same as any other JVM program, which means JNI, JNA, or FFM.

from graal.

HashMap and some other Java types are banned from use in certain contexts under optimizing Truffle runtimes

I was not talking about java.util.HashMap. You can change my HashMap.java example to SportCar.java.

The "out of the box" solution is JNI, which has been supported since very early Java (like 1.1 iirc).

I disagree with you. JNI is for calling C/C++ code from Java, through calling native keyword methods. It does not address my use-case.

@sgammon I would LOVE to see an example (talk won't get us too far) for the simple class below:

public class SportCar {

    public void turnOnEngine() {
        System.out.println("Engine is on!");
    }

    public native static SportCar newInstance();

}

How do I compile the Java code below with NativeImage and then proceed to use my SportCar from a regular Java code, which of course won't have the SportCar class in its classpath, in other words, SportCar will come only from the native compiled code, not from the classpath (i.e. not from any ByteCode).

I think the only way to do that is to introduce a Car.java interface, that both the NativeImage and the subsequent Java code that uses SportCar will have on its classpath. But besides that, compiling Java code with NativeImage to be called from another Java code is a mystery to me. This is very different than JNI.

from graal.

@letmedrawit I think what you mean is "thank you." Three people have already tried to help you including me, who explained your options in great detail.

There is of course the JNI Invocation API (linked earlier) which has a sample embedded.

Good luck

from graal.

How would I distribute the binaries of SportCar.java while still allowing customers to use my class? It is not possible to do that with GraalVM.

It is, via the exact process described. If the problem is passing a Java object over a JNI border, there are other answers for that.

from graal.

JNI isn't the best solution and I certainly wish there was an easier way. I think an easier way could be possible given that GraalVM is building from bytecode and ultimately it needs those same interfaces to accomplish this.

However, as it is now, a shared native library is just that: a native library. You can accomplish this with a lot of glue via the JNI Invocation API, and it will work, but no question it can be a pain. Static JNI is possible (although undocumented) for actually embedding the library in the final binary, but as far as I can tell that's the opposite of what you are asking for.

from graal.

@sgammon I think what I'm trying to do is just not possible. It is outside the scope of GraalVM, which is totally fine. I just wish there was an easy way of doing that, but there isn't for now. Totally fine. Thanks for the help and sorry to be a pain :)

from graal.

Well, I think that link might have what I need. I missed that, sorry. As long as I can also do the same thing for non-static methods. I'll give it a try and report back here => https://www.graalvm.org/latest/reference-manual/native-image/native-code-interoperability/JNIInvocationAPI/

Note that this document describes the opposite of what is commonly done via JNI: usually low-level system operations are implemented in C and invoked from Java using JNI.

That's what I need: the opposite what is commonly done via JNI

Thanks and sorry again. I'll give it a try and report back here if it works.

from graal.

@letmedrawit No worries, but I think one or both of us are still confused.

The premise is wrong. I do NOT have any native shared library to build. I have a pure Java library I would like to distribute in binary format.

What do you mean by "binary format?" As I see it, there are two things you could mean:

1. "Binary format" as in ELF or Mach-O, I.e. native code for a given operating system. These are typically referred to as "shared libraries." You can make shared libraries with many languages including C/C++ or Java through Native Image.

2. "Binary format" as in JVM class bytecode, which is how compiled classes are usually distributed in the JVM ecosystem. I.e. within JARs, JMODs, or other packaging.

I assume you mean 1 because 2 is already pretty familiar to most Java devs and isn't any different on GraalVM anyway as compared to other JDKs.

Am I understanding correctly?

If so, a shared library is an option, which would be an ".so" (as you put it) on Linux, a ".dll" on Windows, and so on.

from graal.

I also just wanted to volunteer that this topic was very confusing to me, as a Java dev, when I first waded through it. So I think it is completely normal to reject JNI for this purpose; I get it. But you may want to take a second look.

from graal.

@letmedrawit take a look at this sample; hopefully it can be helpful to you. The README explains how to try it. The build happens via Gradle, but the process is the same in Maven etc.

from graal.

(Ah, looks like we're on the same page now. Notifications were lagged; that's great to hear.)

from graal.