This repository hosts the public releases of Acme traces from the Shanghai AI Lab, encompassing workloads spanning from March 2023 to August 2023. We encourage anyone to use the traces for academic purposes, and if you had any questions, feel free to send an email to us, or file an issue on Github.

Furthermore, we have conducted a thorough analysis of the Acme workloads, detailed in our NSDI '24 paper titled Characterization of Large Language Model Development in the Datacenter.

Link:

The main trace characteristics, dataset structure and schema are:

• Full Dataset size: 80GB (on HuggingFace)
• Dataset size: 109MB
• Duration: 6 months
• Number of independent GPU clusters: 2
• Total number of jobs: 880,740
• Total number of GPU jobs: 470,497

📦AcmeTrace
 ┣ 📂data
 ┃ ┣ 📂job_trace 
 ┃ ┃ ┣ 📂trace_previous_work              (Prior job traces for comparison)
 ┃ ┃ ┃ ┣ 📜helios_trace.csv
 ┃ ┃ ┃ ┣ 📜xxx.csv
 ┃ ┃ ┣ 📜trace_kalos.csv                  (Job trace file, collected from scheduler)
 ┃ ┃ ┗ 📜trace_seren.csv
 ┃ ┣ 📂utilization
 ┃ ┃ ┣ 📂ipmi                             (Power of different server models in Seren, collected from IPMI)
 ┃ ┃ ┃ ┣ 📜CPU_D_Power.csv
 ┃ ┃ ┃ ┣ 📜GPU_AB_Power.csv
 ┃ ┃ ┃ ┗ 📜GPU_C_Power.csv
 ┃ ┃ ┣ 📂kalos                            (Resource utilization logs, collected from DCGM & Prometheus)
 ┃ ┃ ┃ ┣ 📜DRAM_ACTIVE.csv
 ┃ ┃ ┃ ┣ 📜xxx.csv
 ┃ ┃ ┣ 📂seren
 ┃ ┃ ┃ ┣ 📜DRAM_ACTIVE.csv
 ┃ ┃ ┃ ┣ 📜xxx.csv
 ┃ ┃ ┣ 📂util_pkl                         (Processed pickle files for plotting)
 ┃ ┃ ┃ ┣ 📜gpu_power_kalos.pkl
 ┃ ┃ ┃ ┣ 📜xxx.pkl
 ┃ ┣ 📜cluster_summary.csv
 ┃ ┣ 📜generate_utilization_pkl.ipynb     (Parse utilization files and generate pickles)
 ┃ ┗ 📜utils.py
 ┣ 📂figure                               (Examples of trace visualization)
 ┃ ┣ 📜bar_job_state.pdf
 ┃ ┣ 📜xxx.pdf
 ┣ 📜LICENSE.txt
 ┣ 📜README.md
 ┗ 📜analysis.ipynb                       (Scripts for plotting)

Provides rich information on all jobs submitted to scheduler in each cluster.

• trace_seren.csv Example

• trace_kalos.csv Example

Only in Kalos:

1. A job can end up with one of five statuses: (1) COMPLETED: it is finished successfully; (2) CANCELLED: it is terminated by the user; (3) FAILED: it is terminated due to internal or external errors; (4) TIMEOUT: the execution time is out of limit; (5) NODE_FAIL: it is terminated due to the node crash. TIMEOUT and NODE_FAIL are very rare in our traces, and are regarded as failed in our analysis.
2. Calculated from the difference between end_time and start_time. (Unit: seconds)
3. Calculated from the difference between start_time and submit_time. (Unit: seconds)
4. Calculated from the product between duration and gpu_num.

Cluster resource utilization monitoring data, collected from DCGM, IPMI and Prometheus.

• NODE_CPU_UTILIZATION.csv Example