openucx / openhpca Goto Github PK

The default execution mode would only run the tests required to generate the metrics.
The long execution mode would run all the tests.

I encountered an issue with the SMB msgrate benchmark when testing using the long version of the benchmark run. It seems that the execution hangs early on and the job is eventually killed by slurm when the time limit is reached.

Looking at the script generated by the Go infrastructure, one problem is likely to be that the benchmark specific parameters are not currently passed in. However, something else may also be going on. Adding the correct flags did not fix the issue in my case. Execution seems to be hanging at, or about, MPI_thread_init(), for yet undiscovered reasons.

The above fail to execute as because the necessary binary arguments are not passed in via generated slurm scripts.

@BrodyWilliams discovered that make init is relying on openhpca_setup but that binary is generated when executing make install. I think make init should ensure openhpca_setup is correctly generated when necessary.

I see the following error sometimes after all jobs are done. Unclear at this point if a job failed or if the compilation of results is just facing a problem:

ERROR: unable to display results: undefined data

The WebUI code in doing too much of the processing to display the results, which increase the likeliness to create gaps between the WebUI and the text result file.

There is a need to run benchmarks suites like OSU in a automated manner. So having a run option to choose a specific benchmark suite would be useful.

Taylor suggested to provide a profiling capability that would give more details about overlapping when running real applications as well as, if possible, the potential for more overlapping. This could be initially investigated within the collective profiler (https://github.com/gvallee/collective_profiler). These two projects share some of the underlying building blocks so, if relevant, it should be possible to integrate both projects.

gvallee/go_hpc_jobmgr#6

I'm trying to run some experiments with the time-driven execution model inside the overlap_XYZ collectives, and for all of them, regardless of MPI library, I get the following error (see attached):

This error occurs no matter how high I set OPENHPCA_OVERLAP_MAX_NUM_ELTS, though for some reason, I'm able to get per-message-size latencies if I turn on the calibration feature (not documented, see overlap.h for OPENHPCA_OVERLAP_CALIBRATION). Because of this, I'm not able to see the final output generated for the Time-Driven model (work injected, latency, %-overlap, etc.)

In this particular case, I'm running overlap_igather with the following command: mpirun -np <ntasks> --hostfile hostfile /path/to/overlap_igather after exporting OPENHPCA_OVERLAP_MAX_NUM_ELTS

What can I do to fix this?
Thank you!

Make sure the text file with the results includes all the metrics. I have seen a potential issue during a test on a new platform.

https://github.com/yqin/osu-micro-benchmarks is not available anymore and breaks everything.

After running the suite, it would be helpful to display a report about the runs. For instance, it would be nice to have an explicit report when OPENHPCA_OVERLAP_MAX_NUM_ELTS needs to be set to increase the default value used by the overlap benchmarks.