mlcommons / cm4mlops Goto Github PK

We were asked to help students run MLPerf inference benchmark at the Student Cluster Competition'24 and automate their submission and grading via the MLCommons CM automation framework.

Current plan is to use MLPerf inference Stable Diffusion benchmark with Stability AI’s Stable Diffusion XL model (2.6 billion parameters) and COCO data set. This popular model is used to create compelling images through a text-based prompt.

We must check the following:

• Check current CM workflows to run reference MLPerf SD benchmark
• Check CM workflows to run optimized MLPerf SD benchmark v4.0
  • Intel
  • Nvidia
• Check if support for AMD GPUs can be provided
• Check how to support multi-node inference
• Prepare tutorial about MLPerf, loadgen, this benchmark and CM
• Check MLCommons Croissant format for the dataset?
• Automate submission and grading
  • Need to agree how to report accuracy
  • We may train a smaller model to analyze produced images
  • Create live scoreboard (W&B?)

Aggregating tasks to improve CM script automation and CM scripts based on user feedback (this ticket is being gradually updated and tasks resolved based on our bandwidth and engineering resources):

Automation

• Add --silent / -s mode to avoiding printing CM workflow execution
• Add cm_min_version check for automation and scripts
• Check current dumping of all versions of all dependencies Commit
• Check readme generation with all deps Commit
  • Add --version when available Commit
• Prepare tmp_run_final_script.sh and stop to let user run it manually without CM if needed
• Generate sample Docker container during --repro
• Add cfg to load default env, state and keys for all CM scripts
  • cm set cfg default --key.cm-script.silent --key.cm-script.env.CM_SUDO="no" ...
  • Add silent mode as default
  • Support add/update cfg keys ...
• Add cm status to show version, paths to current repositories and if the version is up-to-date (useful for virtual environments)
• Add to meta where to report errors if not default repository (for MLPerf inference -> report https://github.com/mlcommons/inference)
• Add proper version detection for cuDNN

Documentation

• Check individual documentation and input description for all main CM scripts
• Add individual tests for all main CM scripts

Tutorials

Need to explain how to extend CM scripts, add new ones, unify inputs, etc)

• Prepare tutorial about CM basics and CM scripts
• Prepare tutorial about CM automation for basic inference
• Prepare tutorial about CM automation for MLPerf loadgen
• Prepare tutorial about CM automation for MLPerf inference
• Prepare tutorial about CM automation for ABTF
• Prepare tutorial about CM automation for SCC'24

Longer term

• Use logging instead of print
• Refactor and simplify CM script automation - it was heavily prototyped and it is stable but the implementation can be considerably improved and simplified
• Refactor docker container generation and execution - the prototype implementation is very complex and can be dramatically simplified

running bdist_wheel
Traceback (most recent call last):
 File "/home/ebay/CM/repos/local/cache/2bfde55101034352/pytorch/setup.py", line 1288, in <module>
   main()
 File "/home/ebay/CM/repos/local/cache/2bfde55101034352/pytorch/setup.py", line 1239, in main
   setup(
 File "/home/ebay/projects/devops/mlperftest/cm/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
   return distutils.core.setup(**attrs)
 File "/usr/lib/python3.10/distutils/core.py", line 148, in setup
   dist.run_commands()
 File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
   self.run_command(cmd)
 File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
   cmd_obj.run()
 File "/home/ebay/CM/repos/local/cache/2bfde55101034352/pytorch/setup.py", line 744, in run
   with concat_license_files(include_files=True):
 File "/home/ebay/CM/repos/local/cache/2bfde55101034352/pytorch/setup.py", line 722, in __enter__
   create_bundled(os.path.relpath(third_party_path), f1,
 File "/home/ebay/CM/repos/local/cache/2bfde55101034352/pytorch/third_party/build_bundled.py", line 42, in create_bundled
   collected = collect_license(d)
 File "/home/ebay/CM/repos/local/cache/2bfde55101034352/pytorch/third_party/build_bundled.py", line 20, in collect_license
   raise ValueError('could not identify license file '
ValueError: could not identify license file for third_party/opentelemetry-cpp/tools/vcpkg/ports/hungarian

CM error: Portable CM script failed (name = install-pytorch-from-src, return code = 256)

cm-run-script-input.json
cm-run-script-info.json

• https://github.com/mlcommons/ck/tree/master/.github/workflows
• https://github.com/mlcommons/cm4mlops/tree/master/.github/workflows

We may rely on some extra functionality in CM in scripts so we need to check min CM version and suggest to upgrade it ...

The procedure to install the latest docker (CPU & CUDA) is somewhat tricky for different OS. I suggest to create a script to aggregate all this knowledge in one place. It will simplify using CM workflows with MLPerf and other benchmarks via Docker.

We need to continue improving universal benchmarking and optimization capabilities in CM for different OS and hardware targets:

• For compiled code (C/C++ ...) we improve the following CM scripts

  • CM scripts:
    • compile-program
    • benchmark-program
    • benchmark-program-mlperf
  • TBD
    • support for gprof/oprofile/hardware counters
    • universal support to pin threads (numactl)
    • expose internal profiling info from ML frameworks and run-times if/when available (onnx, TFLite ...)
  • Sample apps:
    • Corner detection (C)
    • MLPerf ONNX C++ harness
    • MLPerf TFLite C++ harness

• For Python:

  • create CM script with a python package to collect various profiling info (memory utilization, etc) particularly to analyze ML/AI models (was asked by ABTF).
  • Collect function-level profiling

• Add support for universal performance analysis to CM experiment:

  • Aggregate profiling from multiple runs and perform stat analysis (variation, min/max, phases, etc)
  • Visualize experiments

We should extend our universal compile and benchmark scripts to support cross-platform compilation and execution (requested by several MLCommons workgroups):

• compile-program
• benchmark-program
• benchmark-program-mlperf

As a test example, we can compile and run program for Android or some SSH-based platform.

We should add get-target-device CM to define target platform capabilities (env, config and tools) similar to how I did it in the original CK framework and add it to the build/run CM script.

We had feedback from some MLPerf users that they do not have sudo access while most of their system deps are already installed. In such case, we should have a flag and env var to skip all SUDO/system installations ...

We had to remove configs with private keys from the CM get-rclone script.

We need to add proper support to handle such private keys. We can use the new CM cfg automation or create a "get-rclone-config" script.

TBD: should brainstorm more.

Hi @arjunsuresh . Let sync how to do that . Thanks!