Running benchmarks with varying OCAMLRUNPARAM about sandmark HOT 5 CLOSED

So another use case which is a bit more general, but I think we should include in any fix here, is being able to set arbitrary environment variables for the benchmark run.

For example, an experiment people wanted to be able to run recently was glibc malloc vs mimalloc vs jemalloc vs tcmalloc vs etc... In theory it might be as simple as specifying LD_PRELOAD in the environment, which if we can somehow just set from within the run_config.json then that would be good.

from sandmark.

Does the following syntax in an ocaml-versions/4.10.0+stock.json file be sufficient?

{
  "url" : "https://github.com/ocaml/ocaml/archive/4.10.0.tar.gz",
  "configure" : "-q",
  "runparams" : "v=0x400",
  "experiments" : {
      "test1" : "OCAMLRUNPARAM='s=1M'",
      "test2" : "OCAMLRUNPARAM='s=2M'",
      "test3" : "OCAMLRUNPARAM='s=4M'",
      "test4" : "OCAMLRUNPARAM='s=8M'"
      "testj" : "LD_PRELOAD=/path/to/jemalloc.so OCAMLRUNPARAM='v=0x400'"
   }
}

The output will be a different _results (test1, test2, etc.) folder for each experiment.

from sandmark.

If we have stuff in the compiler configuration to define the environment variables for the run of the benchmarks, this is going to get confusing fast, IMHO. For example, taskset or chroot needs to go into the wrappers block of run_config.json but the environment variables would end up in the ocaml-versions directory. I would have thought all of those things from a user perspective look like the environment within which a benchmark runs.

With this in mind, is there anything wrong with trying to put this into the wrappers block of the run_config.json file?

Maybe something like:

    {
      "name": "orun.2M.blah",
      "environment": {"OCAMLRUNPARAM":"s=2M", "ENV_X":"blah"},
      "command": "orun -o %{output} -- taskset --cpu-list 5 %{command}"
    }

I'm open to other suggestions, but I'm keen to try and group all the shell environment variables along with taskset, chroot, etc. into one place.

from sandmark.

Agreed on keeping them out of the compiler definition @ctk21 , I too think that would be confusing.

I'm a little concerned about putting them on the wrapper too, as we'd probably end up with a blowup of wrappers if you wanted to test a bunch of environments.

Could we have a separate environments section in the run config, which for each environment had a name (which could be concatenated on to the end of the wrapper name for runs) and a list of environment variables? I'm thinking we could also extend that functionality to allow for other things in the future, like being able to specify a script to run pre/post.

The question is though, if we add that then how do you specify which environment you want if you only want to do a single run? I guess there can be a default environment so the behaviour is the same as it is now but if you supplied "all" it would just do all environments for the wrapper you've chosen?

from sandmark.

orun actually has support for OCAMLRUNPARAM:
https://github.com/ocaml-bench/sandmark/blob/master/orun/orun.ml#L265

But, we don't pass it in run-time:
https://github.com/ocaml-bench/sandmark/blob/master/Makefile#L125

I tested setting OCAMLRUNPARAM in the above line in the Makefile, and the values take effect during execution.

So, the idea is to specify the list of environment options for the different experiments with a labelled name, entirely in one location, so that the "Executing benchmarks ..." in the Makefile can be iterated for the required number of experiments, passing the respective value for OCAMLRUNPARAM, all in one go.

If a different set of experiments are needed, it will just involve creating another ocaml-verions/file.json file, with the specific OCAMLRUNPARAM environment settings. Yes, the configuration is in two places, but, eventually with Sandmark 2.0, the idea is to move to a single file.

Option A
If it is fine to have a large number of wrappers, how should one invoke one or many chosen wrappers to execute the benchmarks? I am fine with this approach.

Option B

Could we have a separate environments section in the run config, which for each environment had a name (which could be concatenated on to the end of the wrapper name for runs) and a list of environment variables?

Can you please give an example to show the preferred configuration?

from sandmark.