NCCL WARN NET/OFI Only EFA provider is supported about aws-ofi-nccl HOT 2 CLOSED

@mkserge As you noted, aws-ofi-nccl plugin is supported only on p3dn.24xlarge and p4d.24xlarge instances. Also, EFA does't support p3.16xlarge too (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types).

Since you only want to do intra-node communication, I can advise the following:

1. Remove EFA installation from your docker image as this isn't a supported platform type.
2. Use NCCL's socket interface directly rather than using plugin. (You can change LD_LIBRARY_PATH so that NCCL is not able to dynamically load the plugin)

I hope that helps.

from aws-ofi-nccl.

Hi Rashika,

Thank you for your response and apologies for the delay.

I went ahead and removed all the EFA-related parts from the Dockerfile in aws/deep-learning-containers repository and rebuild the image from scratch and it seems to be working now. Thank you for your advice!

It seems a bit odd that the AWS DLC ship with NCCL with EFA support only. It looks like that essentially prevents one from doing distributed training with NCCL backend on anything other than those two instances, unless one is willing to build the base images themselves. I opened a related issue in aws/deep-learning-containers repository to get some feedback on this.

from aws-ofi-nccl.