Comments (5)
So another use case which is a bit more general, but I think we should include in any fix here, is being able to set arbitrary environment variables for the benchmark run.
For example, an experiment people wanted to be able to run recently was glibc malloc vs mimalloc vs jemalloc vs tcmalloc vs etc... In theory it might be as simple as specifying LD_PRELOAD
in the environment, which if we can somehow just set from within the run_config.json
then that would be good.
from sandmark.
Does the following syntax in an ocaml-versions/4.10.0+stock.json
file be sufficient?
{
"url" : "https://github.com/ocaml/ocaml/archive/4.10.0.tar.gz",
"configure" : "-q",
"runparams" : "v=0x400",
"experiments" : {
"test1" : "OCAMLRUNPARAM='s=1M'",
"test2" : "OCAMLRUNPARAM='s=2M'",
"test3" : "OCAMLRUNPARAM='s=4M'",
"test4" : "OCAMLRUNPARAM='s=8M'"
"testj" : "LD_PRELOAD=/path/to/jemalloc.so OCAMLRUNPARAM='v=0x400'"
}
}
The output will be a different _results (test1
, test2
, etc.) folder for each experiment.
from sandmark.
If we have stuff in the compiler configuration to define the environment variables for the run of the benchmarks, this is going to get confusing fast, IMHO. For example, taskset
or chroot
needs to go into the wrappers
block of run_config.json
but the environment variables would end up in the ocaml-versions
directory. I would have thought all of those things from a user perspective look like the environment within which a benchmark runs.
With this in mind, is there anything wrong with trying to put this into the wrappers
block of the run_config.json
file?
Maybe something like:
{
"name": "orun.2M.blah",
"environment": {"OCAMLRUNPARAM":"s=2M", "ENV_X":"blah"},
"command": "orun -o %{output} -- taskset --cpu-list 5 %{command}"
}
I'm open to other suggestions, but I'm keen to try and group all the shell environment variables along with taskset
, chroot
, etc. into one place.
from sandmark.
Agreed on keeping them out of the compiler definition @ctk21 , I too think that would be confusing.
I'm a little concerned about putting them on the wrapper too, as we'd probably end up with a blowup of wrappers if you wanted to test a bunch of environments.
Could we have a separate environments section in the run config, which for each environment had a name (which could be concatenated on to the end of the wrapper name for runs) and a list of environment variables? I'm thinking we could also extend that functionality to allow for other things in the future, like being able to specify a script to run pre/post.
The question is though, if we add that then how do you specify which environment you want if you only want to do a single run? I guess there can be a default environment so the behaviour is the same as it is now but if you supplied "all" it would just do all environments for the wrapper you've chosen?
from sandmark.
orun
actually has support for OCAMLRUNPARAM:
https://github.com/ocaml-bench/sandmark/blob/master/orun/orun.ml#L265
But, we don't pass it in run-time:
https://github.com/ocaml-bench/sandmark/blob/master/Makefile#L125
I tested setting OCAMLRUNPARAM in the above line in the Makefile, and the values take effect during execution.
So, the idea is to specify the list of environment options for the different experiments with a labelled name, entirely in one location, so that the "Executing benchmarks ..." in the Makefile can be iterated for the required number of experiments, passing the respective value for OCAMLRUNPARAM, all in one go.
If a different set of experiments are needed, it will just involve creating another ocaml-verions/file.json
file, with the specific OCAMLRUNPARAM environment settings. Yes, the configuration is in two places, but, eventually with Sandmark 2.0, the idea is to move to a single file.
Option A
If it is fine to have a large number of wrappers, how should one invoke one or many chosen wrappers to execute the benchmarks? I am fine with this approach.
Option B
Could we have a separate environments section in the run config, which for each environment had a name (which could be concatenated on to the end of the wrapper name for runs) and a list of environment variables?
Can you please give an example to show the preferred configuration?
from sandmark.
Related Issues (20)
- Report hardware resources in parallel architectures using `hwloc`
- Integrate SPECpower benchmarks and measurements in Sandmark
- Multicore parallel benchmarks support for 4.14 HOT 2
- Fix Uuidm.create, Stdlib.Printf.kprintf and ocaml_deprecated_auto_include compiler warnings
- `mtime.1.2.0` fails to build with 5.1.0+trunk HOT 1
- Add MaPLe benchmarks to Sandmark HOT 1
- graph500 failing HOT 1
- Add benchmarks from frama-c Open Source case studies
- Include pausetimes in jupyter notebook HOT 1
- Remove the old implementations of graph500
- Update Sandmark trunk to build for 5.2.0 HOT 1
- A number of parallel benchmarks seem to be broken HOT 2
- Include percentiles in the pausetimes bench data HOT 2
- Round off errors for mean and max latencies HOT 2
- Unpin `runtime_events_tools` and pin `ppxlib` to a version compatible with OCaml 5.1 and 5.2 HOT 2
- Get latency and throughput information from the same benchmark runs
- Running parallel benchmarks seems to fail sometimes because chrt permissions are lost?
- Benchmark runs seem to be failing due to base.v0.16.3 install failure
- Benchmark runs seem to be failing due to changes on trunk
- Remote access to Turing broken HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sandmark.