Integrate dotnet-compile-native CLI command with E2E experiences for CoreRT development about corert HOT 19 CLOSED

Chatted with @schellap and came to this set of requirements to light up this scenario:

1. Enable easy acquisition of the dotnet cli
2. Testing of Csharp to Native ( "dotnet compile --native" )
3. Testing of Managed Dll to Native (" dotnet compile-native hello.dll ....")
4. Ability to patch in local CoreRT bits

Feel free to share any thoughts
/cc @gkhanna79 @schellap

from corert.

We dont have support to build (3) above. Other than that, it looks good.

from corert.

Just making each bullet point a bit detailed:

1. Get CoreRT toolchain and its dependencies as well for each platform (and may be flavor -- dbg/rel?). This restoration should be seamless with dev mode pluggability.
2. Compile CS: Include CPP compilation mode and RyuJIT compilation options. Additionally, an option for custom code generators as well. i.e., coming from a different package name other than protojit. LLILC may eventually get plugged in.
3. Compile Managed-EXE: directly through the toolchain with CPP and RyuJIT options. This is needed to run CoreCLR tests. See corert/tests/runtest.cmd which first compiles using CSC and then passes on the exe to dotnet-compile-native.bat
   • Compile Managed-DLL: Eventually we would compile these similar to NGEN, but that capability doesn't exist as of now.
4. Ability to plug in a development mode CoreRT bits, ProtoJit/LlilcJit bits, ObjWriter bits.

In summary, the current experience offered by dotnet-compile-native.bat should be a subset.

from corert.

@schellap
How do you envision plugging custom code generators? Is it planned for ILToNative to support this?

from corert.

Currently, ILToNative has two supported code generators. RyuJIT by default and -cpp option to produce IL to CPP files. ILToNative loads protojit.dll that is specified in the system %path% variable. If you see dotnet-compile-native.bat file, you'd notice we fixup the path for ILToNative to load protojit.dll.

As far as dotnet-cli is concerned, the codegen package name can be specified dynamically and it needs to be in the path for ILToNative to load. We will make ILToNative load some other DLL name based on env variable or command line option, but as long as the specified DLL is in the path, ILToNative should be able to pick it up. @gkhanna79, any opinions?

from corert.

We will make ILToNative load some other DLL name based on env variable or command line option, but as long as the specified DLL is in the path

Tweaking PATH to make things load is a hack - it should not be a shipping solution. The shipping solution should be to load the JIT via LoadLibrary(absolute path), if it is not next to ILToNative.exe.

from corert.

@jkotas, I agree. @brthor, from your perspective, we will support it as an option that allows to specify codegen load path.

from corert.

@schellap
For custom code generators, is the requirement to only produce a Intermediate output or to produce a native executable?

Will future code generators be producing a consistent output type? Currently different modes in ILToNative are producing different outputs (ex. cpp/obj). If this stays the case then we need some custom steps (per OS) to then produce a native executable. Then it seems that we would need an additional pluggability point for handling the steps from the Code Generator output to Native.

Thoughts?

from corert.

The folks should not ever worry about which code generator to use. There should be a default one (per target OS/platform) that we will focus on and that will just work.

The options to override the code generator are primarily a testing hook. Make it extensible just enough for that. If some folks will find these testing hooks useful - that's ok, but there is no point in trying to make it infinitely general. The really advanced use cases are going to skip dotnet compile -native completely and invoke the underlying commands like csc.exe or ILToNative.exe directly.

from corert.

Yeah, someone could come along and say I want to convert IL to Java or Objective C and produce native code from there. LLILC JIT will have very similar interface as protojit. These are the basic requirements as of now and we should not worry about the outside cases for now.

IMO, that would be a nice to have support but not an immediate requirement.

from corert.

Thanks for the feedback @schellap and @jkotas!

For the purposes of coding this right now can I assume ILToNative will support the cmd line args, "--codegenpath" and "--objgenpath" ?

from corert.

I believe objgenpath does not need to be configurable/pluggable, @jkotas, do we need this capability if we'll emit different formats or this is fully encapsulated by LLVM?

ILToNative will support --codegenpath.

from corert.

objgenpath should not need to be configurable.

What is the value of codegenpath going to be? Where is it going to come from?

from corert.

If we decide to have values for --mode that map to different non-CPP codegenerators, codegenpath will point to the location where the code-generator binary will be found. LLILC and ProtoJIT are two examples I can think of - we need to define this formally if we expect community to plugin a new code-generator that implements the JIT-EE interface and native file format.

from corert.

What is the value of codegenpath going to be? Where is it going to come from?

My understanding is as follows:

For the mainline scenario, we would specify the protojit package as ILToNative dependency and it will be placed along side the ILToNative.exe by the DNU restore operation. ILToNative will be specified a codegenpath or simple name (because it is side-by-side) of protojit.dll.

For the scenario targeting protojit development, i.e., to test their bits with ILToNative we need a pluggable JIT with a codegen package path specified to CLI. ILToNative will be specified the path to this JIT as codegenpath.

If we want to switch to llilc JIT, same as above, ILToNative will load a different JIT with the path specified in codegenpath.

As far as dotnet-cli is concerned, this custom codegen package is specifiable to the CLI. The JIT name is also optionally specifiable to CLI or otherwise it will look for protojit.dll in these custom codegen packages.

from corert.

@gkhanna79 @schellap dotnet-compile-native is included in the CoreRT test workflow now.
Can we close this out?

from corert.

Assigning to @schellap to track adding --ilcsdkpath to the E2E flow in CoreRT repo then it can be closed out.

from corert.

@schellap Was this completed as part of your ILCompiler.SDK change?

from corert.

We can close this one out as it is better to not specify custom AppDep path and get corert/cli out of sync. It is however useful for testing purposes.

from corert.