Comments (5)
π
Ha! The lack of benchmarking tools in Python is among the bane of my existence... so much that one of my long term plans/ideas is taking reusable parts of benchee and putting them into a reusable binary and then just reimplementing the runner for different languages.
Ok, sorry.
Is it possible? Yes! You can totes just call System.cmd
Should you do it? Probably not.
- That adds whatever overhead System.cmd has to your measurements
- it also includes the startup time of the python interpreter (which is non negligible)
- it also includes warmup and benchees warmup executions won't do you any good here
- if you want to benchmark CLI executions, I almost hate to admit, but there is a better tool: https://github.com/sharkdp/hyperfine
for 2. + 3. I go into more detail here: https://pragtob.wordpress.com/2017/08/29/careful-what-you-measure-2-1-times-slower-to-4-2-times-faster-mjit-versus-truffle-ruby/ ("What time are we measuring?" section
Hope this helps!
Ah, one more: A very simple benchmarking implementation isn't too hard, see for instance this: https://github.com/PragTob/rubykon/tree/main/lib/benchmark that's less than 200loc for what I think of as a passable and minimally viable benchmarking library :)
from benchee.
My use cases take relatively long times on the order of a minute, so the overhead is negligible for me.
benchees warmup executions won't do you any good here
Why not?
hyperfine
It doesnβt support peak memory consumption.
PS: Will the memory of these forked processes be benchmarked correctly with benchee?
from benchee.
Ah.
- Warmup won't do you any good, as you will start the process again with each iteration - this resets the "warmup" of any runtime, it'll JIT again
- weird that hyperfine doesn't do memory measurement, you can get peak memory usage by using
time -v command
on a unix system (might be /usr/bin/time -v on mac). Benchee also won't give you memory measurements here, as benchee measures the memory of the running elixir/erlang process and not of the operating system level process spawned bySystem.cmd
from benchee.
I am not using a JITed language here, so that doesn't matter. But the inability to measure the memory of forked processes makes benchee unsuitable. Can't you add an option to measure those as well?
I am current using time
, but it is pretty bare-bones. I had to write the logic of running the measurements multiple times and aggregating the results myself.
from benchee.
even in a non JITed language warmup matters for many scenarios. Erlang was non JIT'ed for the majority of benchee's life time. I still implemented it before even the first release (iirc).
Measuring the memory of an externally spawned process is way different from what benchee does. I don't think I'll ever add this, as benchee just isn't the tool for the job here. I wouldn't even know how to do it properly. Before we go there, there are many things to improve around the memory measurement benchee has (it only measures the one process you spawned, not other processes that process may spawn or other processes in the system for instance)
from benchee.
Related Issues (20)
- Sugguestion: Express time in `h:m:s` HOT 4
- Promote use of keywords lists instead maps for job list HOT 3
- Is it worth releasing a new version? HOT 2
- Is there an option to try all possible scenarios once to know early if some would raise ? HOT 2
- Suggestion: Option to kill long-running scenarios early HOT 2
- Question: Benchmark function operations ? HOT 1
- Warn if the benchmarking function is evaluating HOT 2
- Benchee throws an exception when running in the Livebook-App HOT 3
- Benchee warning when used in mix task HOT 2
- Unpredictable experiment order in output HOT 1
- Elixir provide HOT 1
- Built in outlier detection
- suggestion HOT 1
- (Protocol.UndefinedError) protocol List.Chars not implemented for {:error, :bad_name} HOT 5
- High memory consumption on some cases HOT 8
- Top level functions slower than module functions in Elixir <1.16 HOT 7
- Avoid handing all of inputs to every scenario process
- Add Solaris Support to Benchee.System HOT 3
- Benchmarking between BEAM versions HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchee.