a numerical bench: Gram matrix initialization in parallel about sandmark HOT 28 CLOSED

Thanks @UnixJunkie. @Sudha247 will integrate the Multicore OCaml version of this benchmark into sandmark.

from sandmark.

@UnixJunkie : I see that this implementation of gram-matrix uses batteries and batteries doesn't work with Multicore yet. We will have to remove batteries' dependency to build this with Multicore OCaml.

from sandmark.

Ok, I'll get rid of the batteries dependency and let you know.
Should be quick.

from sandmark.

By the way, can you open a bug against batteries?
If there is a problem to fix, we should know about this problem in the first place.
https://github.com/ocaml-batteries-team/batteries-included/issues

from sandmark.

@Sudha247 I'm done, there is no more dependency to batteries in my proposed benchmark. If multicore ocaml manages to beat parmap's parallelization performance, please do drop me an e-mail. :)

from sandmark.

GramMatrix is now included in Sandmark: #100. @Sudha247 can you post the performance comparison between multicore and the other parallel versions?

from sandmark.

I am interested to see the exact command being run to launch the Gram matrix bench using multicore-OCaml.
My current test:

~/src/sandmark/_opam/4.06.0/bin/orun -o test.bench -- taskset --cpu-list 0-15 ~/src/sandmark/_build/default/benchmarks/multicore-grammatrix/grammatrix.exe

just runs a sequential program.

from sandmark.

These are the numbers for grammatrix_multicore.exe on 4.06.1+multicore:

from sandmark.

@Sudha247 what is the command that you are using to get those results?

from sandmark.

The table should include a speedup column (sequential_time / multi-thread_time).

from sandmark.

@UnixJunkie This is the command used to run the multicore version

~/src/sandmark/_opam/4.06.1+multicore+parallel/bin/orun -o ../../grammatrix_multicore.16.orun.bench -- taskset --cpu-list 2-13,16-27 chrt -r 1 ./grammatrix_multicore.exe 16

16 is the number of cores here which can be adjusted.

To run the parallel benchmarks, you can use the run_all_parallel.sh file.

from sandmark.

Here are the results I get with parmap:

#cores parmap_t(s) speedup efficiency
1 57.83 1.00 1.00
2 37.93 1.52 0.76
4 24.94 2.32 0.58
8 16.79 3.44 0.43
12 14.93 3.87 0.32
16 13.45 4.30 0.27

from sandmark.

Your results converted to speedup:

#cores multicore_t(s) speedup efficiency
1 96.73 1.00 1.00
2 74.35 1.30 0.65
4 44.62 2.17 0.54
8 30.74 3.15 0.39
12 28.99 3.34 0.28
16 27.52 3.52 0.22

from sandmark.

Maybe your multi-thread implementation has a load balancing problem.
I guess compute_gram_mat is not constant time (though I have no idea
about what 'Domain.Sync.poll();' is doing).
The job computing the first line of the Gram matrix (which has N elements) should also compute
the last line of the Gram matrix (it has only 1 unknown elements).
The job computing the second line of the Gram matrix (N-1 elements) should also
compute the one before last line of the Gram matrix (it has 2 unknown elements).
etc.

Parmap does load balancing when it is used correctly, so we don't need to be too smart
when implementing parallel things with Parmap.

from sandmark.

@Sudha247 can you post just the time numbers for multicore and parmap that were obtained on our benchmarking server. I assume #99 (comment) and #99 (comment) were obtained on different machines? And include the speedup as well compared to a sequential baseline?

from sandmark.

@kayceesrk your assumption about different machines is correct (I am still unable to run the multicore-ocaml version of the bench...)

from sandmark.

Indeed this might not be the optimal multicore version of this benchmark. On our benchmarking server, the following numbers were obtained.

ConcMinor (concurrent minor collector) is 4.06.1+multicore which can be found here and ParMinor (parallel minor collector) can be found here here. The parmap version was built with 4.06.1 base compiler.

@UnixJunkie do your parmap numbers account for the overhead caused by reading data from the input file?

from sandmark.

I just time the initialization of the Gram matrix:

let () = Gc.full_major () in
      let curr_dt, curr_matrix =
        Utls.wall_clock_time (fun () ->
            compute_gram_matrix style ncores csize samples
          ) in

Note that in your results, up to 12 cores, Parmap is still better.

from sandmark.

@Sudha247 if you manage to scale better, Gram matrix initialization is a useful real-world problem.
Feel free to write the algorithm in which ever way is the most efficient for multicore-OCaml.

from sandmark.

Also, the bench should be run several times for each multic-core config, avg and stddev computed.

from sandmark.

The parmap version was built with 4.10.0 base compiler.

We shouldn't benchmark against two different versions of the compilers. parmap version should also be 4.06.1. Is parmap version using batteries? If so, that must be ported to use whatever multicore is using.

from sandmark.

Parmap does not depend on batteries.

from sandmark.

We shouldn't benchmark against two different versions of the compilers.

Noted, thanks! I have updated my original comment with the numbers for 4.06.1.

from sandmark.

I've redone the multicore version with load balancing. Here are the results:

Running time (seconds)

Baseline = 92.55 seconds

Speedup

Speedup graph is here

from sandmark.

Note that if you play with chunksize to get better performance with multicore-OCaml,
you should also play with it for Parmap and Parany, to see which chunksize is the best for them.

I was also busy. :)

Parany has a new branch called sendmsg. The version on this branch
is significantly faster than before (but only works on Linux for the moment).
Also, I updated the code in
https://github.com/UnixJunkie/gram-matrix-bench
so that the Parmap version is more efficient now.

from sandmark.

I did a bit of experimentation with chunk sizes and it looks like a chunk size of 3 is the most efficient for parmap and parany. For multicore, it is set to 16, though going lower does not impact performance. I ran the benchmark on a 56 core machine.

I ran the benchmarks with the latest updates. parany is performing worse than before.

Graph is here

I'll consider this issue closed for now since we have the Gram Matrix implementation in the sandmark suite :-)

from sandmark.

Yes, and it looks like a clear win for multicore-OCaml.
I am impressed; nice work.

from sandmark.

FTR, the results of my bench (just Parmap Vs. Parany) can be seen
in the picture in the README.md file there:
https://github.com/UnixJunkie/gram-matrix-bench

from sandmark.