typelead / eta-benchmarks Goto Github PK

A set of benchmarks to aid in performance optimisation for the Eta compiler.

Shell 0.36% Haskell 61.18% Makefile 0.65% C 2.25% Java 0.27% TeX 1.02% Mathematica 0.05% C++ 0.04% Assembly 0.13% Clean 0.12% Fortran 2.14% Roff 31.78%

eta haskell functional-programming lazy-evaluation performance

eta-benchmarks's People

Contributors

Stargazers

Watchers

Forkers

agocorona rolandtritsch

eta-benchmarks's Issues

Whats next?

Hi @rahulmutt,

with the latest merge we are at a point where we can reliably build and run the benchmarks.

The loose ends are ...

investigate and fix that last/remaining tests that fail
investigate and fix the problem(s) with the (failing) output checks
investigate and fix all the new problems (e.g. NPEs) that come with 0.8.6b5

... but I am not sure, if this is time well spend, because right now for me the biggest gap is that we are running the benchmarks and we are getting a lot of data, but no insight.

To fix this, I am proposing to add an ability to write the benchmark results to a database so that we can query the database to compare two runs.

This would allow us to compare versions of eta. Lateron we could add tests to compare eta with scala and haskell.

WDYT?

(Some) Tests are failing with 0.8.6.5

They fail with a NPE (e.g. shootout/expert), but pass with 0.8.6.4.

To make this work I have updated init.sh and used etlas select 0.8.6b4 to select a/the right version.

This works locally, but when I run it on the docker container on circleci it fails with ...

etlas: Unable to find eta-0.8.6.4 in any of your configured binary indices.

Any idea why it fails on/with the docker container?

Is this the right/best way to define/lock-down the eta version to use?

Investigate if/why/how it is correct that getArgs returns [""] when no parameters are passed to the test(s)

Take a look at spectral/expert and remove the case on [""] to see the problem.

spectral/para gives truncated output

The output of spectral/para appears to truncate parts of the output arbitrarily. Investigate.

real/anna throws an exception

	Suppressed: eta.runtime.exception.EtaException: 
Syntax error: unexpected token "p9" on line 110

		at base.ghc.TopHandler.runMainIO2(TopHandler.hs)
		at base.ghc.TopHandler$runMainIO2.apply1V(TopHandler.hs)
		at eta.runtime.exception.Exception.catch_(Exception.java:166)
		at main.Main.main3(Main.hs)
		at main.Main.DZCmain(Main.hs:158)
		at main.Main$DZCmain.applyV(Main.hs:158)
		at eta.runtime.stg.Closures$EvalLazyIO.enter(Closures.java:104)
		at eta.runtime.stg.Capability.schedule(Capability.java:174)
		at eta.runtime.stg.Capability.scheduleClosure(Capability.java:119)
		at eta.runtime.Runtime.evalLazyIO(Runtime.java:170)
		at eta.runtime.Runtime.main(Runtime.java:163)
		at eta.main.main(Unknown Source)
		at com.typelead.EtaBenchmark.benchmark(EtaBenchmark.java:45)
		at com.typelead.generated.EtaBenchmark_benchmark_jmhTest.benchmark_ss_jmhStub(EtaBenchmark_benchmark_jmhTest.java:780)
		at com.typelead.generated.EtaBenchmark_benchmark_jmhTest.benchmark_SingleShotTime(EtaBenchmark_benchmark_jmhTest.java:716)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
		at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at java.lang.Thread.run(Thread.java:745)
	Caused by: eta.runtime.exception.EtaException: 
Syntax error: unexpected token "p9" on line 110

		at base.ghc.Err.error(Err.hs:35)
		at main.MyUtils.myFail(MyUtils.hs:14)
		at main.Parser2.$LrJ1Hlvl(Parser2.hs)
		at main.Parser2.$wlvl(Parser2.hs)
		at main.Parser2.paParse1(Parser2.hs)
		at main.Parser2.paParse_get_parse(Parser2.hs:248)
		at main.Parser2.$wpaParse(Parser2.hs:553)
		at main.Main$sat_sI46.thunkEnter(Main.hs)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at ghc_prim.ghc.CString$unpack_s6XT.applyN(CString.hs:113)
		at ghc_prim.ghc.CString$sat_s6XY.thunkEnter(CString.hs:114)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at base.ghc.io.handle.Text.$wa7(Text.hs:586)
		at base.ghc.io.handle.Text.hPutStr2(Text.hs:553)
		at main.Main.main1(Main.hs)
		at main.Main$main1.applyV(Main.hs)
		at eta.runtime.exception.Exception.catch_(Exception.java:129)
		... 24 more

Have eta-bench output GC and JIT stats

spectral/circsim is failing

Output:

FAILED STDOUT
WANTED: [[F,F,F,F,F,F,F,F],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[...
GOT: [[F,F,F,F,F,F,F,F],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[T,T,T,T,T,T,T,T],[...

Running the benchmark fails with "No such file"

When installing/cloning this from scratch, running it fails, because Build.hs needs eta on the path (which is not necessarily the case) to do/run eta --numeric-version (in Build.hs; line 151).

Options to fix this ...

Put a requirement into the README to make sure that eta is on the path (and document a/the best way to put eta on the path (right now I copy it manually from ~/.etlas/binaries/... to ~/.local/bin)).
Change the way Build.hs is looking for the version number

Not sure what the best/right way is to fix this.

Thoughts?

Revisit: spectral/mandel: Disable skip-check again

spectral/mandel produces a different stderr every time it runs.

To fix/workaround this we enabled skip-check (in single-bench), when we run the tests/for all tests.

Skipping the checks for all tests is sub-optimal. Would suggest to revisit this.

One idea would be to make skipping the checks a parameter for single-bench and only skip the check for the tests that need to skip the check (right now spectral/mandel and real/compress).

spectral/treejoin throws irrefutable pattern failed exception

	Suppressed: eta.runtime.exception.EtaException: spectral/treejoin/Main.hs:115:7-35: Irrefutable pattern failed for pattern (f1 : ~(f2 : _))

		at base.ghc.TopHandler.runMainIO2(TopHandler.hs)
		at base.ghc.TopHandler$runMainIO2.apply1V(TopHandler.hs)
		at eta.runtime.exception.Exception.catch_(Exception.java:166)
		at main.Main.main4(Main.hs)
		at main.Main.DZCmain(Main.hs:114)
		at main.Main$DZCmain.applyV(Main.hs:114)
		at eta.runtime.stg.Closures$EvalLazyIO.enter(Closures.java:104)
		at eta.runtime.stg.Capability.schedule(Capability.java:174)
		at eta.runtime.stg.Capability.scheduleClosure(Capability.java:119)
		at eta.runtime.Runtime.evalLazyIO(Runtime.java:170)
		at eta.runtime.Runtime.main(Runtime.java:163)
		at eta.main.main(Unknown Source)
		at com.typelead.EtaBenchmark.benchmark(EtaBenchmark.java:45)
		at com.typelead.generated.EtaBenchmark_benchmark_jmhTest.benchmark_ss_jmhStub(EtaBenchmark_benchmark_jmhTest.java:780)
		at com.typelead.generated.EtaBenchmark_benchmark_jmhTest.benchmark_SingleShotTime(EtaBenchmark_benchmark_jmhTest.java:716)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
		at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at java.lang.Thread.run(Thread.java:745)
	Caused by: eta.runtime.exception.EtaException: spectral/treejoin/Main.hs:115:7-35: Irrefutable pattern failed for pattern (f1 : ~(f2 : _))

		at base.ghc.io.handle.FD$sat_sIOYF.apply1V(FD.hs)
		at eta.runtime.exception.Exception.catch_(Exception.java:166)
		at base.ghc.io.handle.FD.openFile1(FD.hs:274)
		at main.Main.main1(Main.hs)
		at main.Main$main1.applyV(Main.hs)
		at eta.runtime.exception.Exception.catch_(Exception.java:129)
		... 24 more
	Caused by: eta.runtime.exception.EtaException: spectral/treejoin/Main.hs:115:7-35: Irrefutable pattern failed for pattern (f1 : ~(f2 : _))

		at base.ghc.Exception.$Lr8LTPlvl(Exception.hs)
		at base.ghc.Exception.throw1(Exception.hs)
		at base.control.exception.Base.irrefutPatError(Base.hs:396)
		at main.Main$main3.thunkEnter(Main.hs)
		at eta.runtime.thunk.CAF.evaluate(CAF.java:35)
		at main.Main$ds2_sEU0.thunkEnter(Main.hs)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at eta.runtime.thunk.SelectorUpd.thunkEnter(SelectorUpd.java:19)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at base.java.StringBase.$wgo(StringBase.hs:70)
		at base.java.StringBase.$wtoJString(StringBase.hs:69)
		at base.ghc.io.FD.openFile1(FD.hs:172)
		at base.ghc.io.handle.FD.openBinaryFile3(FD.hs:178)
		at base.ghc.io.handle.FD$sat_sIOXL.applyV(FD.hs:145)
		at eta.runtime.exception.Exception.catch_(Exception.java:129)

imaginary/integrate is failing

The output is:

FAILED STDOUT
WANTED: 0.0
GOT: --0.0

spectral/treejoin: Fix null pointer exepction.

Revisit: real/compress: hGetContents: invalid argument (invalid byte sequence)

real/compress likes to fail with an exception (hGetContents: invalid argument (invalid byte sequence).

To make real/compress compile and run again hSetBinaryMode stdout needs to be False (in Main.hs) and skip-check needs to be enabled (in single-bench).

Sooner or later we want to run all test without skipping the checks again. For that to work we need to make sure we write the correct binary data to stdout (otherwise the checks will fail).

Selector Thunk bug in spectral/multiplier

	Suppressed: java.lang.NoSuchFieldError: x1
		at main.Main.$wmultsys(Main.hs:149)
		at main.Main$ds_sOVB.thunkEnter(Main.hs)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at eta.runtime.thunk.SelectorUpd.thunkEnter(SelectorUpd.java:19)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at base.ghc.Base.map(Base.hs:850)
		at main.Main$sat_sOVR.thunkEnter(Main.hs)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at base.ghc.List.null(List.hs:115)
		at base.ghc.List$null.apply1(List.hs:115)
		at base.ghc.Base$sat_sAH6.thunkEnter(Base.hs:851)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at main.Main.$LrNXBgo5(Main.hs)
		at main.Main.trans(Main.hs:421)
		at main.Main.$wtraceMult(Main.hs:121)
		at main.Main$sat_sP07.thunkEnter(Main.hs)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at ghc_prim.ghc.CString$unpack_s6YI.applyN(CString.hs:138)
		at ghc_prim.ghc.CString$sat_s6YN.thunkEnter(CString.hs:139)
		at eta.runtime.thunk.UpdatableThunk.evaluate(UpdatableThunk.java:24)
		at base.ghc.io.handle.Text.$wa7(Text.hs:586)
		at base.ghc.io.handle.Text.hPutStr2(Text.hs:553)
		at main.Main.main1(Main.hs)
		at main.Main$main1.applyV(Main.hs)
		at eta.runtime.exception.Exception.catch_(Exception.java:129)
		at main.Main.main8(Main.hs)
		at main.Main.DZCmain(Main.hs:95)
		at main.Main$DZCmain.applyV(Main.hs:95)
		at eta.runtime.stg.Closures$EvalLazyIO.enter(Closures.java:104)
		at eta.runtime.stg.Capability.schedule(Capability.java:174)
		at eta.runtime.stg.Capability.scheduleClosure(Capability.java:119)
		at eta.runtime.Runtime.evalLazyIO(Runtime.java:170)
		at eta.runtime.Runtime.main(Runtime.java:163)
		at eta.main.main(Unknown Source)