Currently, a simple program like

main = putStrLn "Hello World!"

generates a ~30MB JAR file. An estimated 75-80% of that amount is dead code. The goal is to workout a proguard config file that works and include proguard as a step in EPM for optimised builds.

Motivation

After discussion with @aloiscochard and his eagerness to provide support from the JVM assembler side, I have decided to depend on his kuna project instead of hs-java.

Implementation

1. Observe the structure of kuna and how the currently existing parts of the code generator can be rewritten in kuna style. This exploration should happen on a separate branch called wip/kuna.
2. Remove the dependency of the CodeGen monad on the GenerateIO monad from the hs-java library.
3. Rewrite the existing parts of the codegen with the new kuna-based monad.

https://github.com/ghc/packages-containers

Depends on #36, #37

2.7.1.0 - https://github.com/ghc/packages-unix
2.3.1.0 - https://github.com/ghc/packages-Win32

Unify these two packages into an 'os' package as Java abstracts from the OS.

We would like something like

data {-# CLASS "java.util.ArrayList" #-} ArrayList a = ArrayList (Object# (ArrayList a))

foreign import java unsafe "add" add :: a -> Java (ArrayList a) ()
foreign import java unsafe "get" get :: Int -> Java (ArrayList a) a

This is very close to becoming a reality - we just need to add a new primitive unsafeClassCast# :: Object# a -> Object# b that will generate a checkcast bytecode. This primitive is useful in general for manual casting of Java objects inside of Haskell code.

The current naive implementation of jar linking extracts all the class files and puts them all into a single jar. Using a streaming abstraction should make this faster. The CodeGen monad needs to be slightly updated to support streaming (write classfiles as soon as the code is generated for a given closure).

https://github.com/ghc/packages-time

Depends on #37

https://github.com/ghc/packages-filepath

Motivation

Currently, GHCVM generates a new inner class for almost every top-level declaration and hence there will be a lot of files of the form Module$toplevel.class generated at the same level of the source module or wherever the target directory is specified using the -o flag. It would be better to stick them all into a JAR file.

Implementation

1. The codeGen function in GHCVM.CodeGen.Main will return a set of Class values that can be serialized into a Java class. The next step after serialization should be to package them all up into a JAR file.

The differences between GHCVM and GHC should be documented comprehensively in the wiki.

Motivation

Currently, ghcvm has a dependency on libzip (transitive from the hs-java dependency) which is making it difficult to build (see #7).

Implementation

Write a utility module that can JAR class files together, making use of the pure Haskell library zip.

Commit: cfd5914

Haskell side:

{-# LANGUAGE MagicHash #-}
module Export where

import Primes
import GHC.Base

data {-# CLASS "hello.Export" #-} Export = Export (Object# Export)

foreign export java takePrime :: Int -> Java Export Int

takePrime i = return . fromIntegral $ primes !! i

module Primes where

primes = filterPrime [2..]
  where filterPrime (p:xs) =
          p : filterPrime [x | x <- xs, x `mod` p /= 0]

Java side:

import hello.Export;

public class HelloWorld {
    public static void main(String[] args) {
        String v = Integer.toString(new Export().takePrime(1));
        System.out.println(v);
    }
}

Error:

java.lang.VerifyError: Operand stack underflow
Exception Details:
  Location:
    hello/Export.takePrime(I)I @31: getfield
  Reason:
    Attempt to pop empty stack.
  Current Frame:
    bci: @31
    flags: { }
    locals: { 'hello/Export', integer }
    stack: { }
  Bytecode:
    0x0000000: b800 182a bb00 1a59 b200 20bb 000b 591b
    0x0000010: b700 23b7 0026 b800 2ab4 002e b800 32b4
    0x0000020: 0035 c000 0bb4 0039 ac                 

https://github.com/ghc/packages-pretty

Depends on #37

@sibi::jane { ~/github/ghcvm-hackage/test/array-0.5.1.1 }-> ghcvm-pkg list
/home/sibi/.ghcvm/package.conf.d
   array-0.5.1.1
   base-4.8.2.0
   ghc-prim-0.4.0.0
   integer-0.5.1.0
   rts-0.1.0.0
/home/sibi/.ghcvm/package.conf.d
   array-0.5.1.1
   base-4.8.2.0
   ghc-prim-0.4.0.0
   integer-0.5.1.0
   rts-0.1.0.0
/home/sibi/.ghcvm/package.conf.d
   array-0.5.1.1
   base-4.8.2.0
   ghc-prim-0.4.0.0
   integer-0.5.1.0
   rts-0.1.0.0
/home/sibi/.ghcvm/package.conf.d
   array-0.5.1.1
   base-4.8.2.0
   ghc-prim-0.4.0.0
   integer-0.5.1.0
   rts-0.1.0.0

Probably this happens because of multiple installation (i.e cleaninstall.sh) ?

While foreign imports allow us to call java functions from within Haskell in a type-safe way, foreign exports allow us to export a Haskell function that can be called as a java function. More specifically, to accommodate the way Java frameworks are managed, we should allow existing classes to be subclasses within Haskell itself and implement interfaces. The alternative would be to roll out your own subclass and make each method call an exported Haskell function which would be unnecessary boilerplate.

Example of instance methods:

data {-# CLASS "mypackage.SomeClass" #-} SomeClass = SomeClass (Object# SomeClass)

someMethod :: Int -> Java SomeClass ()
someMethod = ...

foreign export java someMethod :: Int -> Java SomeClass ()

This will create a new class with name "SomeClass.class" and with a method called "void someMethod(int)" which takes an int and returns void.

add :: Int -> Int -> Int
add = ...

foreign export java add :: Int -> Int -> Int

This will statically export "int add(int, int)" as a static method of the current class.

module Main where

main = do
  print $ 2 + 15
  print $ 49 * 100

This program prints out 17 and then hangs.

Hi Rahul,

I was wondering about the naming and the scope you want to cover here, the reason I ask is because with a name with ghcvm I would expect it to potentially try to target different vm, or to implement a vm itself.

Wouldn't be ghcjvm more appropriate? (it also reflect the naming used by the JS backend)

Cheers!

https://github.com/ghc/packages-deepseq

Depends on #36

Some Java frameworks use annotations as triggers for bytecode instrumentation and other purposes. It would be useful to be able to export Haskell functions with certain annotations.

The main requirement for support is to use the existing features of GHC to convey the annotation information. If that is too restrictive, we'll have to consider a syntactic extension.

There is one tricky aspect: specifying values for annotations. This is not THAT common so it's fine if we leave it for some time.

Open to proposals on how the syntax should look. Preferably with examples of usage in commonly used Java frameworks (such as Spring).

We should build a forked version of Cabal built with GHC 7.10.3 that will help us compile the rest of the packages easily. It should be customised to deal with the specifics of GHCVM.

Steps to do the port:

1. etlas get process-1.6.1.0
2. cd process-1.6.1.0
3. etlas build

• This will show you that directory >=1.1 && <1.4, filepath >=1.2 && <1.5, unix >=2.5 && <2.8 are needed. As a general rule, you should comment out the unix and Win32 dependency when doing ports since it's hard to shim the full package on the JVM (uses lots of platform-dependent C FFI calls) and we want to keep packages as portable as possible. We can offer platform-dependent versions for performance later on.

4. Open the process.cabal file and comment out all platform-dependent stuff, including:

• the c-sources:, includes:, and install-includes: fields (i.e. anything that contains .c or .h files).
• any conditional dependencies based on platform (i.e the entire if os(windows) clause should be commented out.)
  You can use -- for single-line comments.

5. Change the build-type: field to Simple as that's the only build type supported by etlas, for now.
6. Run etlas install directory-1.3.1.0 (the future versions are not supported for now) followed by etlas install --dependencies-only to install the rest.
7. Run etlas build.
8. You'll find lots of errors about missing modules. Remove those modules and you'll find now that functions are missing. Now your task is to study what's happening around that missing function, and figure out which Java API you need to use to substitute that functionality.
9. Keep fixing stuff, run etlas build, and keep going until the package builds successfully.

Bindings to Java's ProcessBuilder API most likely have to be made.

See the instructions here to create and submit the patch once you're done.

The run script doesn't work on Ubuntu. Find an alternative way to generate the script.

When the user defines two top-level functions that are equivalent modulo case, they will currently get a NoClassDef error when running the program. We should stop this prematurely after renaming, informing the user that he shouldn't do this. For now we'll mention it clearly in the docs.

The current instance definitions for Class involve boilerplate.

data {-# CLASS "java.lang.Object" #-} Object = Object (Object# Object)

instance Class Object where
  obj = Object
  unobj (Object x) = x

This would be great to reduce to

data {-# CLASS "java.lang.Object" #-} Object = Object (Object# Object)
  deriving Class

Currently, the following function declarations

module Test.Main where
putStrLn = ...

putstrln = ...

generate a class test/Main$putStrLn.class and test/Main$putstrln.class but JARs are case-insensitive so both of them clash and only one arbitrary class file gets chosen while the other disappears, yielding a NoClassDef error at runtime. The current naming scheme is GHC's Z-encoding, but we need to extend it to make sure two functions names that are equivalent modulo case produce names that are not equivalent modulo case.

The function GHCVM.CodeGen.Name.nameText is the central function which regulates naming for closure identifiers. Looking for a naming scheme that keeps the names clean.

https://github.com/ghc/packages-bytestring

Depends on #37

ghcvm --make SomeClass.class SomeOtherClass.java SomeJar.jar SomePure.hs

Should convert all the given files to class files accordingly and properly put them in the resulting jar. Currently, it's partially implemented (should work in batch mode -c) but needs to be tweaked for --make mode. In particular, we need to make sure we set the -classpath flag of javac properly, taking into account references to the module closures compiled in the same batch.

Implementation Sketch:
Separate out .java files and compile them only after all other compilation units and include all the compiled .jars as dependencies.

module Main where

main = do
  print $ take 10 (repeat 5)

This gives:

Exception in thread "main" java.lang.IllegalAccessError
    at base.ghc.List$repeat.enter(Unknown Source)
    at ghcvm.runtime.apply.StgFun.apply(StgFun.java:116)
    at ghcvm.runtime.apply.Apply$8.enter(Apply.java:75)
    at main.Main$main2.thunkEnter(Unknown Source)
    at ghcvm.runtime.thunk.StgIndStatic.enter(StgIndStatic.java:30)
    at ghcvm.runtime.stg.StgClosure.evaluate(StgClosure.java:20)
    at base.ghc.List$take.enter(Unknown Source)
    at ghcvm.runtime.apply.StgFun.apply(StgFun.java:189)
    at ghcvm.runtime.apply.Apply$11.enter(Apply.java:104)
    at main.Main$main1.thunkEnter(Unknown Source)
    at ghcvm.runtime.thunk.StgIndStatic.enter(StgIndStatic.java:30)
    at ghcvm.runtime.stg.StgClosure.evaluate(StgClosure.java:20)
    at base.ghc.Show$showListzuzu.enter(Unknown Source)
    at base.ghc.Show$zdLrFHWzdcshowList22.enter(Unknown Source)
    at ghcvm.runtime.apply.StgFun.apply(StgFun.java:189)
    at ghcvm.runtime.apply.ApPP.stackEnter(ApPP.java:19)
    at ghcvm.runtime.stg.StackFrame.enter(StackFrame.java:43)
    at ghcvm.runtime.stg.StackFrame.enter(StackFrame.java:26)
    at ghcvm.runtime.stg.StgContext.checkForStackFrames(StgContext.java:75)
    at base.ghc.io.handle.Text$shoveStringzus7ZZSF.enter(Unknown Source)
    at base.ghc.io.handle.Text$zdLr7V3SwriteBlocks.enter(Unknown Source)
    at base.ghc.io.handle.Text$satzus7ZZVC.enter(Unknown Source)
    at ghcvm.runtime.apply.StgFun.apply(StgFun.java:134)
    at ghcvm.runtime.apply.Apply$9.enter(Apply.java:84)
    at base.ghc.Base$zdLr4JLa3.enter(Unknown Source)
    at ghcvm.runtime.apply.Apply$20.enter(Apply.java:210)
    at ghcvm.runtime.apply.StgPAP.apply(StgPAP.java:46)
    at ghcvm.runtime.apply.ApV.stackEnter(ApV.java:12)
    at ghcvm.runtime.stg.StackFrame.enter(StackFrame.java:43)
    at ghcvm.runtime.stg.StackFrame.enter(StackFrame.java:26)
    at ghcvm.runtime.stg.StackFrame.enter(StackFrame.java:26)
    at ghcvm.runtime.stg.Capability.schedule(Capability.java:245)
    at ghcvm.runtime.RtsScheduler.scheduleWaitThread(RtsScheduler.java:57)
    at ghcvm.runtime.Rts.evalLazyIO(Rts.java:91)
    at ghcvm.runtime.Rts.hsMain(Rts.java:36)
    at ghcvm.main.main(Unknown Source)

module Main where

main = do
  print $ take 10 (cycle [1,2,3])

It works for take 1, 2 & 3, beyond that it hangs. It seems to be a problem with the cycle function.

The GHCVM installation process currently copies over

ghc-usage.txt
settings
include/ghcversion.h
include/ghcplatform.h

This is only a temporary solution to quickly get access to GHC settings, and there are much better ways of solving the problem, like refactoring the main ghcvm executable project.

We should remove the dependency on the location of the GHC installation directory and provide a GHCVM-specific solution instead.

https://github.com/ghc/packages-directory

Depends #39 & #38

https://github.com/ghc/packages-binary

Depends on #43, #42, #36

Currently, anonymous inner classes are used to declare RtsFuns. Instead, we should used named inner classes so that debugging becomes clearer. An example is eta.runtime.apply.Apply which contains a lot of them which are frequently used. It's easier to decipher what Apply$ApPFast is rather than Apply$8.

For some reason, take returns [] when compiled with -O0 but the correct answer when compiled with -O2. This deviation should be checked. The current workaround is to use the elementary implementation of take.

Motivation

A suggestion was given on reddit about supporting the C FFI for greater library support on Hackage.

My reservations on this matter are:

1. Using JNI locks you onto the platforms that the C library supports.
   Comment: After further thought, most server-side applications in production will typically not shift between architectures/hardware very frequently and during the time they migrate then can link a different version of the library that supports that platform. @mgsloan also notes that some of the Hackage libraries make use of cross-platform C libraries, so this may not be a serious issue.
2. Too many context switches.
   Comment: In order to call to the C FFI, first you must call a native Java method (which is a Java FFI call) and then the JVM does a context switch to the native world. So essentially, you're switching from Haskell world -> Java world -> Native world and then you have to follow that sequence of switches in reverse. I expect this to be costly, but we can test it out and see what happens. Moreover, there may be cases where it is actually faster to use the Java method that performs the same task if it exists in some library. This needs to be checked.

Implementation

The RTS needs no special support other than that of Java FFI. The entire implementation will be in the codegen.

1. When a foreign import declaration with the ccall calling convention is used

foreign import ccall "someheader.h somefunc" somefunc :: Int -> Int

an appropriate native method declaration is made with the Z-encoded Haskell name in the declaration.

1. A field is generated which generates a function closure of matching arity which upon entry should call the generated native method, taking in the appropriate arguments and casting as appropriate.
2. A static initializer block is generated to load the dynamic library containing the native method implementation.
3. After code generation, another step is performed using javah to generate the header file for the class and the native method declaration and to generate a .c file that will call the C function mentioned above in quotes.
4. After the C file generation, gcc should be run to compile the generated C file into a .dll (if windows) or .so (if linux). This part will be tricky handling all the platforms.

There are minor differences between GHC and GHCVM and those minor differences must be accounted for to allow libraries on Hackage to compile. To consult with every package maintainer about coming up with the fixes themselves is a time-consuming task as well as waiting for them to release a new version that will contain GHCVM support via a conditional #ifdef __GHCVM_VERSION__ check.

I propose that we:

• introduce a new repo called ghcvm-hackage which contains information on package, version, and how to patch that specific package version so that GHCVM can compile it. (This will literally be a git patch file).
• Add a new feature in cabalvm that will clone the ghcvm-hackage repo locally and keep updating it. Whenever a user invokes cabalvm install [package], the package will be looked up in ghcvm-hackage and the source distribution will be patched with the patch file just before cabalvm starts the build.

The resulting flow:

1. A user tries installing a package and finds that the build fails.
2. He files an issue to the ghcvm-packages repository
3. People who want to help out will:

• download that package from hackage locally
• make the necessary changes to fix the build
• submit a PR with the patch file to the ghcvm-hackage repo
• the end-user will be able to successfully install the package upon another invocation of cabalvm install

Looking forward to feedback.

For the initial start, the test can be just a simple hello world and the prime number example which is documented in the wiki.

The code for return points comes within the same closure rather than a separate block. This prevents us from being able to safely context switch which relies on the property that a TSO's state of execution can be restored purely from its stack.

The solution is to create a StackFrame with stackEnter() overriden with the code for the return point. In cgCase, we need to generate a new StackFrame and then push that stack frame before we evaluate the scrutinee.

The main disadvantage is that this generates a lot more class files, but that's a necessary sacrifice for implementing concurrency/parallelism, an essential feature. The excess class files can be overcome with invoke dynamic if found to be a problem.

This requires adding a bit of logic in StgClosure.enter that checks if context switch needs, and if so, stores the current closure's entry on the stack and safety. The latter bit is tricky and needs to handle the different types of closures. A virtual function saveContext() can de declared and overriden for the different closure types.

Has a dependency on #22.

It'd be nice to detect performance regressions as we make new commits and fix them on the spot. There should be some integration into shake that uses JMH, since Java benchmarking is quite tricky due to JIT.

Hi there,
Thanks for this cool project. I tried to build it on my mac but I got some errors: looks like some dependencies are missing

>./build.sh                                                
mtl-2.2.1: using precompiled package
data-default-0.5.3: configure
monadloc-0.7.1: download
binary-state-0.1.1: download
data-default-0.5.3: build
bindings-libzip-0.10.2: configure
data-default-0.5.3: copy/register
monadloc-0.7.1: configure
monadloc-0.7.1: build
binary-state-0.1.1: configure
binary-state-0.1.1: build
monadloc-0.7.1: copy/register
binary-state-0.1.1: copy/register
Progress: 5/23
--  While building package bindings-libzip-0.10.2 using:
      /Users/prat/.stack/setup-exe-cache/x86_64-osx/setup-Simple-Cabal-1.22.5.0-ghc-7.10.3 --builddir=.stack-work/dist/x86_64-osx/Cabal-1.22.5.0 configure --with-ghc=/usr/local/bin/ghc --with-ghc-pkg=/usr/local/bin/ghc-pkg --user --package-db=clear --package-db=global --package-db=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/pkgdb --libdir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/lib --bindir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/bin --datadir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/share --libexecdir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/libexec --sysconfdir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/etc --docdir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/doc/bindings-libzip-0.10.2 --htmldir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/doc/bindings-libzip-0.10.2 --haddockdir=/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/doc/bindings-libzip-0.10.2 --dependency=base=base-4.8.2.0-bfcc6de9ee2962ae54db40ece3022dc5 --dependency=bindings-DSL=bindings-DSL-1.0.23-75369fdfdd50bafd3a7650315e217654
    Process exited with code: ExitFailure 1
    Logs have been written to: /Users/prat/prt/ghcvm/.stack-work/logs/bindings-libzip-0.10.2.log

    Configuring bindings-libzip-0.10.2...
    setup-Simple-Cabal-1.22.5.0-ghc-7.10.3: The pkg-config package 'libzip'
    version ==0.10.* is required but it could not be found.
Executable named rtsbuild not found on path: ["/Users/prat/prt/ghcvm/.stack-work/install/x86_64-osx/lts-5.11/7.10.3/bin","/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/bin","/Users/prat/.stack/programs/x86_64-osx/ghc-7.10.2/bin/ghc","/Users/prat/.local/bin","/Users/prat/.cabal/bin","/usr/local/bin","/usr/local/sbin","/usr/bin","/bin","/usr/sbin","/sbin","/usr/local/git/bin","/usr/local/munki","/Users/prat/Library/Android/sdk/tools","/Users/prat/Library/Android/sdk/platform-tools","/Users/prat/.cargo/bin","/Users/prat/.cabal/bin"]
Executable named rtsbuild not found on path: ["/Users/prat/prt/ghcvm/.stack-work/install/x86_64-osx/lts-5.11/7.10.3/bin","/Users/prat/.stack/snapshots/x86_64-osx/lts-5.11/7.10.3/bin","/Users/prat/.stack/programs/x86_64-osx/ghc-7.10.2/bin/ghc","/Users/prat/.local/bin","/Users/prat/.cabal/bin","/usr/local/bin","/usr/local/sbin","/usr/bin","/bin","/usr/sbin","/sbin","/usr/local/git/bin","/usr/local/munki","/Users/prat/Library/Android/sdk/tools","/Users/prat/Library/Android/sdk/platform-tools","/Users/prat/.cargo/bin","/Users/prat/.cabal/bin"]

Any suggestion? Thanks again!

It seems that your project is very similar in scope to frege-lang.org. Are you sure it would not be better to contribute to Frege?

Motivation

Currently, the build has a dependency on cpp, the C preprocessor in linux which is making it difficult to build (see #7).

Implementation

Integrate cpphs, a pure Haskell pre-processor into the Shake build system in shake/Build.hs.

https://github.com/ghc/packages-array

primes = 2:3:filter isPrime [5,7..] :: [Int]
isPrime x = all (/= 0) . map (rem x) . takeWhile ((<= x) . (^2)) $ primes
main = print . length . takeWhile (<= 2^24) $ primes

$ java -classpath sievej.jar ghcvm.main
Exception in thread "main" java.lang.NullPointerException
    at ghcvm.runtime.stg.Capability.schedule(Capability.java:296)
    at ghcvm.runtime.RtsScheduler.scheduleWaitThread(RtsScheduler.java:57)
    at ghcvm.runtime.Rts.evalLazyIO(Rts.java:91)
    at ghcvm.runtime.Rts.hsMain(Rts.java:36)
    at ghcvm.main.main(Unknown Source)

takeWhile (<= 100)

$ java -classpath sievej.jar ghcvm.main
25

Currently when running ghcvm --help after install ghcvm, an error message is given.

$ ghcvm --help
/home/chris/.ghcvm/ghc-usage.txt: openFile: does not exist (No such file or directory)

module Main where

main = do
  print $ boomBangs [13,9]

boomBangs xs = [ if x < 10 then "BOOM!" else "BANG!" | x <- xs, odd x]

Works

module Main where

main = do
  print $ boomBangs [7,9]

boomBangs xs = [ if x < 10 then "BOOM!" else "BANG!" | x <- xs, odd x]

Hangs

module Main where

main = do
  print $ boomBangs [7,13]

boomBangs xs = [ if x < 10 then "BOOM!" else "BANG!" | x <- xs, odd x]

Works

module Main where

main = do
  print $ boomBangs [13,15]

boomBangs xs = [ if x < 10 then "BOOM!" else "BANG!" | x <- xs, odd x]

Hangs

The current implementation of GHCVM is fairly unoptimised. Before pursuing optimisations, it'd be nice to have a set of test suites that discover any breakages along the way. A good way to quickly get a test suite would be to re-use GHC's, but they use Makefiles. To keep the building/testing/benchmarking cross-platform, we should port those Makefiles to shake scripts.