why multi-gpu training slower than single gpu about dali HOT 1 OPEN

Hello @wangdada-love ,

Thank you for the interesting question. Let me answer the 2nd one first. DALI is used in MLPerf competition in the benchmarks posted by NVIDIA. Since MLPerf is all about performance, if the native TF would be faster, we'd be using that one ;) Additionally, we do have multitude of success stories (please refer here) that emphasise how DALI helps in data augmentation.

With regards to your firs question, without some additional details it is hard to tell what's happening. Should you like to diagnose what's happening, I'd like to suggest you two things. First, please look at the output of nvidia-smi and htop and verify if your worker resources are 100% utilized. If they are not it is likely that you need to tune training hyperparameters (e.g. num_threads, batch_size, hw_decoder_load) to fit into multi-GPU environment. Secondly, you may want to profile your training. You can find many resources and tutorials on profiling using Nsight systems. TLDR - you can invoke your training using nsys like this:

nsys profile -o my_profile python train.py

And then use Nsight Systems to open captured profile and look what happened.

from dali.