Support for benchmarking memory usage, etc. about hyperfine HOT 15 OPEN

It would be great to be able to optionally get peak RAM consumption.

from hyperfine.

I'm inclined to revisit this idea. Mostly because we now have the option of running commands without an intermediate shell (-N/--shell=none). I am proposing the following stripped-down feature as a start:

• Limit this functionality to cases where --shell=none is set.
• Measure peak RAM usage using getrusage on Linux. If we do not find a working version for Windows, limit this feature to Linux only for now.
• Do not display peak RAM usage on the terminal. Only add it to the JSON export.

See #321 for an earlier implementation of this feature.

from hyperfine.

Any news?

from hyperfine.

I'm not sure of what @chrish42 had in mind, but peak memory usage is an interesting information. I'm not sure how to implement it, though. If someone gives me some pointers, I might give it a go.

from hyperfine.

Since we use getrusage() the maximum resident set would come for free though:

rusage.ru_maxrss

I can try to bake a patch if adding this would be acceptable.

from hyperfine.

I came across this thread because I used hyperfine several times but now also need the peak memory usage for certain commands.
I found another tool which is comparable to hyperfind and serves me well (Unix only): multitime.

Maybe it's a good alternative for some (like me) or serves as inspiration.

from hyperfine.

I used that /usr/bin/time hack and made something rudamentary that might work for me. I wanted to leave it here for anyone to use or improve on.

# Run hyperfine with output printed to the screen.
# In this example, we're just running it at 5 times minimum.
$ hyperfine --show-output -m 5 \
  `# make a command with just ls` \
  -n 'ls' \
  `# run the time command and redirect ls output to /dev/null. But then redirect time output to stdout.` \
  '/usr/bin/time ls -lh > /dev/null' 2>&1 | \
  perl -lane '
    # capture any maxresident integers
    if(/(\d+)maxresident/){
      # add the maxresident to time and increment the count for later average
      $time += $1; 
      $num++
    }; 
    # If we see hyperfine output then print it untouched
    if(/ Time| Range/){
      print;
    } 
    # When hyperfine is done, then average the memory and print it
    END{
      $avg = int($time/$num); 
      print "Avg memory: ${avg}k";
    }'
  Time (mean ± σ):      10.5 ms ±   4.7 ms    [User: 2.0 ms, System: 7.9 ms]
  Range (min … max):     6.0 ms …  63.6 ms    144 runs
Avg memory: 1441k

from hyperfine.

Thank you very much for your feedback.

This are valuable points, but I don't have any plans to extend this to memory profiling/benchmarking. I think this would turn hyperfine into a much more complicated program (code-wise an possibly usage-wise).

That being said, I'm open to discuss this.

from hyperfine.

I tend to think this better left to (far more advanced) tools like valgrind.

from hyperfine.

I'm going to close this, as I don't have any plans to include memory profiling.

from hyperfine.

It's not that easy.

• We do not run the benchmarked process directly, but through a shell. rusuage would probably show the memory usage of both the shell process and the benchmarked subprocess. We would probably have to calibrate our memory measurement by running the shell without the subprocess first (similar to what we do for timing measurements).
• hyperfine is a cross-platform tool. rusage only works on Unix.
• To what extent would memory usage be supported as an additional metric? Do we want to build averages, stddev, etc. like we do for the time measurements? Do we run the outlier detection on the memory usage samples as well?
• How would this additional information be displayed in hyperfines terminal output?
• How would this additional information be represented in the various export formats?

from hyperfine.

It's not that easy.

* We do not run the benchmarked process directly, but through a shell. `rusuage` would probably show the memory usage of both the shell process and the benchmarked subprocess. We would probably have to calibrate our memory measurement by running the shell without the subprocess first (similar to what we do for timing measurements).

* `hyperfine` is a cross-platform tool. `rusage` only works on Unix.

The performance tools for windows have the same capabilities

* To what extent would memory usage be supported as an additional metric? Do we want to build averages, stddev, etc. like we do for the time measurements? Do we run the outlier detection on the memory usage samples as well?

It would be great to have.

* How would this additional information be displayed in `hyperfines` terminal output?

I would add another (optional) line next to the timings.

* How would this additional information be represented in the various export formats?

I hadn't thought about this, probably a second set of files would be the easiest.

from hyperfine.

This would be amazing!
@sharkdp Hope you are working on it!

from hyperfine.

Valgrind does not support Windows or macOS.

I would love for a portable tool like hyperfine to feature peak memory usage!

The current workaround involves using /usr/bin/time with either -v or -l flags. Annoyingly, the flags are mutually exclusive, depending on the particular UNIX implementation. Those hacks are doable for WSL and certain Cygwin-like environments, but there are very few equivalents for native Windows.

from hyperfine.

I'd be interested in this too

from hyperfine.