Comments (5)
bosilca
commented on June 2, 2024
It looks as if your head node has more interfaces (on different subnets)
than the rest of your cluster. Restrict the TCP BTL (via `—mca
btl_tcp_if_include …) to either a single interface (en0 or eth0) or to a
specific subnet (162.55.0.0/16)
George.
…On Fri, May 3, 2024 at 15:01 rafelamer ***@***.***> wrote:
Hi,
for educational purposes, I run a 3-node cluster with openmpi. My hosts
file is
[mce-eseiaat.com](http://mce-eseiaat.com/) slots=8
[worker1.mce-eseiaat.com](http://worker1.mce-eseiaat.com/) slots=4
[worker2.mce-eseiaat.com](http://worker2.mce-eseiaat.com/) slots=4
The three hosts run Fedora 40 with the packages
openmpi-5.0.2-2.fc40.x86_64 and openmpi-devel-5.0.2-2.fc40.x86_64
installed. I have a simple program mpi02.c:
#include <stdio.h>
#include <mpi.h>
#include <string.h>
/*
MPI_Init - Initializes the MPI execution environment
MPI_Comm_rank - Determines the rank of the calling process in the communicator
MPI_Comm_size - Returns the size of the group associated with a communicator
MPI_Finalize - Terminates MPI execution environment
struct MPI_Status {
int MPI_SOURCE;
int MPI_TAG;
int MPI_ERROR;
};
*/
void main(int argc, char *argv[])
{
char message[20];
MPI_Status status;
int i,rank, size, tag=11;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
/**************************
5.Check the rank, and
**************************/
if(rank == 0)
{
strcpy(message, "Hello,World!");
for (i=1; i<size; i++)
MPI_Send(message, 13, MPI_CHAR, i, tag, MPI_COMM_WORLD);
}
else
{
MPI_Recv(message, 20, MPI_CHAR, 0, tag, MPI_COMM_WORLD, &status);
printf( "Process %d : %.13s\n", rank,message);
}
MPI_Finalize();
}
but when I try to run it with
~$ mpiexec -np 16 --hostfile ~/hosts ./mpi02
I have the followin errors
worker1.mce-eseiaat.com:rank8: Trying to connect from eth0 port 1 (subnet 88.198.157.233/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
worker1.mce-eseiaat.com:rank11: Trying to connect from eth0 port 1 (subnet 88.198.157.233/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
worker2.mce-eseiaat.com:rank13: Trying to connect from eth0 port 1 (subnet 162.55.213.49/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
worker1.mce-eseiaat.com:rank9: Trying to connect from eth0 port 1 (subnet 88.198.157.233/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
worker2.mce-eseiaat.com:rank14: Trying to connect from eth0 port 1 (subnet 162.55.213.49/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
worker1.mce-eseiaat.com:rank10: Trying to connect from eth0 port 1 (subnet 88.198.157.233/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
worker2.mce-eseiaat.com:rank15: Trying to connect from eth0 port 1 (subnet 162.55.213.49/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
worker2.mce-eseiaat.com:rank12: Trying to connect from eth0 port 1 (subnet 162.55.213.49/32) to a node (IP=195.201.223.246/32 TCP=53485 UDP=56497) on a different subnet 195.201.223.246/32
Process 1 : Hello,World!
Process 2 : Hello,World!
Process 3 : Hello,World!
Process 5 : Hello,World!
Process 4 : Hello,World!
Process 6 : Hello,World!
Process 7 : Hello,World!
mce-eseiaat.com:rank0: Trying to connect from eth0 port 1 (subnet 195.201.223.246/32) to a node (IP=88.198.157.233/32 TCP=48455 UDP=41246) on a different subnet 88.198.157.233/32
[worker2:04323] *** Process received signal ***
[worker2:04323] Signal: Aborted (6)
[worker2:04323] Signal code: (-6)
[worker1:03602] *** Process received signal ***
[worker1:03602] Signal: Aborted (6)
[worker1:03602] Signal code: (-6)
[worker1:03602] [ 0] [worker2:04324] *** Process received signal ***
[worker2:04324] Signal: Aborted (6)
[worker2:04324] Signal code: (-6)
[worker1:03601] *** Process received signal ***
[worker1:03601] Signal: Aborted (6)
[worker1:03601] Signal code: (-6)
[worker1:03601] [ 0] /lib64/libc.so.6(+0x40710)[0x7f03dac53710]
[worker1:03601] [ 1] /lib64/libc.so.6(+0x98144)[0x7f03dacab144]
[worker1:03601] [ 2] [worker2:04325] *** Process received signal ***
[worker2:04325] Signal: Aborted (6)
[worker2:04325] Signal code: (-6)
[worker1:03599] *** Process received signal ***
[worker1:03599] Signal: Aborted (6)
[worker1:03599] Signal code: (-6)
[worker2:04324] [ 0] /lib64/libc.so.6(+0x40710)[0x7efdde453710]
[worker2:04324] [ 1] /lib64/libc.so.6(+0x98144)[0x7efdde4ab144]
[worker2:04324] [ 2] [worker1:03600] *** Process received signal ***
[worker1:03600] Signal: Aborted (6)
[worker1:03600] Signal code: (-6)
[worker1:03600] [ 0] /lib64/libc.so.6(+0x40710)[0x7f139ce53710]
[worker1:03600] [worker2:04323] [ 0] /lib64/libc.so.6(+0x40710)[0x7f2e46a53710]
[worker2:04323] [ 1] /lib64/libc.so.6(+0x40710)[0x7f2801e53710]
[worker1:03602] [ 1] /lib64/libc.so.6(+0x98144)[0x7f2801eab144]
[worker1:03602] [ 2] /lib64/libc.so.6(gsignal+0x1e)[0x7f2801e5365e]
[worker1:03602] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7f2801e3b902]
[worker1:03602] [ 4] /lib64/libfabric.so.1(+0xfc7f)[0x7f280140fc7f]
[worker1:03602] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7f2801a4f9b2]
[worker1:03602] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7f2801a74d57]
[worker1:03602] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7f2801a770e1]
[worker1:03602] [ 8] /lib64/libc.so.6(gsignal+0x1e)[0x7efdde45365e]
[worker2:04324] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7efdde43b902]
[worker2:04324] [ 4] /lib64/libc.so.6(gsignal+0x1e)[0x7f03dac5365e]
[worker1:03601] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7f03dac3b902]
[worker1:03601] [ 4] /lib64/libfabric.so.1(+0xfc7f)[0x7f03da20fc7f]
[worker1:03601] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7f03da84f9b2]
[worker1:03601] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7f03da874d57]
[worker1:03601] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7f03da8770e1]
[worker1:03601] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7f03da88d2c1]
[worker1:03601] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7f03da81ff1e]
[worker1:03601] [10] /lib64/libfabric.so.1(+0x60f303)[0x7f03da80f303]
[worker1:03601] [11] /lib64/libfabric.so.1(+0x62b504)[0x7f03da82b504]
[worker1:03601] [12] [worker2:04325] [ 0] [ 1] /lib64/libc.so.6(+0x98144)[0x7f139ceab144]
[worker1:03600] [ 2] /lib64/libfabric.so.1(+0xfc7f)[0x7efddda0fc7f]
[worker2:04324] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7efdde04f9b2]
[worker2:04324] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7efdde074d57]
[worker2:04324] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7efdde0770e1]
[worker2:04324] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7efdde08d2c1]
[worker2:04324] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7efdde01ff1e]
[worker2:04324] [10] /lib64/libfabric.so.1(+0x60f303)[0x7efdde00f303]
[worker2:04324] [11] /lib64/libfabric.so.1(+0x62b504)[0x7efdde02b504]
/lib64/libfabric.so.1(+0x68d2c1)[0x7f2801a8d2c1]
[worker1:03602] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7f2801a1ff1e]
[worker1:03602] [10] /lib64/libfabric.so.1(+0x60f303)[0x7f2801a0f303]
[worker1:03602] [11] /lib64/libfabric.so.1(+0x62b504)[0x7f2801a2b504]
[worker1:03602] [12] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7f28021ebcde]
[worker1:03602] [13] /usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7f2802278158]
[worker1:03602] /lib64/libc.so.6(+0x98144)[0x7f2e46aab144]
[worker2:04323] [ 2] /lib64/libc.so.6(gsignal+0x1e)[0x7f2e46a5365e]
[worker2:04323] [ 3] /lib64/libc.so.6(gsignal+0x1e)[0x7f139ce5365e]
[worker1:03600] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7f139ce3b902]
[worker1:03600] [ 4] /lib64/libfabric.so.1(+0xfc7f)[0x7f139c20fc7f]
[worker1:03600] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7f139c84f9b2]
[worker1:03600] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7f139c874d57]
[worker1:03600] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7f139c8770e1]
[worker1:03600] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7f139c88d2c1]
[worker1:03600] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7f139c81ff1e]
[worker1:03600] [10] /lib64/libfabric.so.1(+0x60f303)[0x7f139c80f303]
[worker1:03600] [11] /lib64/libfabric.so.1(+0x62b504)[0x7f139c82b504]
[worker1:03600] [12] [worker2:04324] [12] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7efdde7ebcde]
[worker2:04324] [13] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7f03dafebcde]
[worker1:03601] [13] /usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7f03db078158]
[worker1:03601] [14] /usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7efdde878158]
[worker2:04324] [14] /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7efdde6e48a8]
[14] /lib64/libc.so.6(abort+0xdf)[0x7f2e46a3b902]
[worker2:04323] [ 4] /lib64/libfabric.so.1(+0xfc7f)[0x7f2e45e0fc7f]
[worker2:04323] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7f2e4644f9b2]
[worker2:04323] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7f2e46474d57]
/usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7f03daee48a8]
[worker1:03601] [15] ./mpi02[0x401265]
[worker1:03601] [16] /lib64/libc.so.6(+0x2a088)[0x7f03dac3d088]
[worker1:03601] [17] [worker2:04324] [15] ./mpi02[0x401265]
[worker2:04324] [16] /lib64/libc.so.6(+0x2a088)[0x7efdde43d088]
[worker2:04324] [17] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7efdde43d14b]
[worker2:04324] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7f139d1ebcde]
[worker1:03600] [13] [worker2:04323] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7f2e464770e1]
[worker2:04323] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7f2e4648d2c1]
[worker2:04323] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7f2e4641ff1e]
[worker2:04323] [10] /lib64/libfabric.so.1(+0x60f303)[0x7f2e4640f303]
[worker2:04323] [11] /lib64/libfabric.so.1(+0x62b504)[0x7f2e4642b504]
[worker2:04323] [12] /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7f28020e48a8]
[worker1:03602] [15] ./mpi02[0x401265]
[worker1:03602] [16] /lib64/libc.so.6(+0x2a088)[0x7f2801e3d088]
[worker1:03602] [17] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7f2801e3d14b]
[worker1:03602] [18] ./mpi02[0x4010c5]
[worker1:03602] *** End of error message ***
[18] ./mpi02[0x4010c5]
[worker2:04324] *** End of error message ***
/lib64/libc.so.6(__libc_start_main+0x8b)[0x7f03dac3d14b]
[worker1:03601] [18] ./mpi02[0x4010c5]
[worker1:03601] *** End of error message ***
/lib64/libc.so.6(+0x40710)[0x7f5736853710]
[worker2:04325] [ 1] /usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7f139d278158]
[worker1:03600] [14] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7f2e46debcde]
[worker2:04323] [13] [worker1:03599] [ 0] [worker2:04322] *** Process received signal ***
[worker2:04322] Signal: Aborted (6)
[worker2:04322] Signal code: (-6)
/usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7f139d0e48a8]
[worker1:03600] [15] ./mpi02[0x401265]
[worker1:03600] [16] /lib64/libc.so.6(+0x98144)[0x7f57368ab144]
[worker2:04325] [ 2] /lib64/libc.so.6(+0x40710)[0x7fde99253710]
[worker1:03599] [ 1] /lib64/libc.so.6(gsignal+0x1e)[0x7f573685365e]
[worker2:04325] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7f573683b902]
[worker2:04325] [ 4] /lib64/libfabric.so.1(+0xfc7f)[0x7f5735e0fc7f]
[worker2:04325] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7f573644f9b2]
[worker2:04325] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7f5736474d57]
[worker2:04325] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7f57364770e1]
[worker2:04325] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7f573648d2c1]
[worker2:04325] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7f573641ff1e]
[worker2:04325] [10] /lib64/libfabric.so.1(+0x60f303)[0x7f573640f303]
[worker2:04325] [11] /lib64/libfabric.so.1(+0x62b504)[0x7f573642b504]
[worker2:04325] [12] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7f5736bebcde]
[worker2:04325] [13] /usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7f5736c78158]
[worker2:04325] [14] /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7f5736ae48a8]
[worker2:04325] [15] ./mpi02[0x401265]
[worker2:04325] [16] /lib64/libc.so.6(+0x2a088)[0x7f573683d088]
[worker2:04325] [17] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7f573683d14b]
[worker2:04325] [18] ./mpi02[0x4010c5]
/lib64/libc.so.6(+0x2a088)[0x7f139ce3d088]
[worker1:03600] [17] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7f139ce3d14b]
[worker1:03600] [18] ./mpi02[0x4010c5]
[worker1:03600] *** End of error message ***
/usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7f2e46e78158]
[worker2:04323] [14] /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7f2e46ce48a8]
[worker2:04323] [15] ./mpi02[0x401265]
[worker2:04323] [16] /lib64/libc.so.6(+0x2a088)[0x7f2e46a3d088]
[worker2:04323] [17] /lib64/libc.so.6(+0x98144)[0x7fde992ab144]
[worker1:03599] [ 2] /lib64/libc.so.6(gsignal+0x1e)[0x7fde9925365e]
[worker1:03599] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7fde9923b902]
[worker1:03599] [ 4] [worker2:04322] [ 0] /lib64/libc.so.6(+0x40710)[0x7f865e253710]
[worker2:04322] [ 1] /lib64/libc.so.6(+0x98144)[0x7f865e2ab144]
[worker2:04322] [ 2] /lib64/libc.so.6(gsignal+0x1e)[0x7f865e25365e]
[worker2:04322] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7f865e23b902]
[worker2:04322] [ 4] /lib64/libfabric.so.1(+0xfc7f)[0x7f865d80fc7f]
[worker2:04322] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7f865de4f9b2]
[worker2:04322] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7f865de74d57]
[worker2:04322] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7f865de770e1]
[worker2:04322] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7f865de8d2c1]
[worker2:04322] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7f865de1ff1e]
[worker2:04322] [10] /lib64/libfabric.so.1(+0x60f303)[0x7f865de0f303]
[worker2:04322] [11] /lib64/libfabric.so.1(+0x62b504)[0x7f865de2b504]
[worker2:04322] [12] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7f865e5ebcde]
[worker2:04322] [13] /usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7f865e678158]
[worker2:04322] [14] /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7f865e4e48a8]
[worker2:04322] [15] ./mpi02[0x401265]
[worker2:04322] [16] /lib64/libc.so.6(+0x2a088)[0x7f865e23d088]
[worker2:04322] [17] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7f865e23d14b]
[worker2:04322] [18] ./mpi02[0x4010c5]
[worker2:04322] *** End of error message ***
/lib64/libfabric.so.1(+0xfc7f)[0x7fde9880fc7f]
[worker1:03599] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7fde98e4f9b2]
[worker1:03599] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7fde98e74d57]
[worker1:03599] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7fde98e770e1]
[worker1:03599] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7fde98e8d2c1]
[worker1:03599] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7fde98e1ff1e]
[worker1:03599] [10] /lib64/libfabric.so.1(+0x60f303)[0x7fde98e0f303]
[worker1:03599] [11] /lib64/libfabric.so.1(+0x62b504)[0x7fde98e2b504]
[worker1:03599] [12] [worker2:04325] *** End of error message ***
/usr/lib64/openmpi/lib/libmpi.so.40(+0x1ebcde)[0x7fde995ebcde]
[worker1:03599] [13] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7f2e46a3d14b]
[worker2:04323] [18] ./mpi02[0x4010c5]
[worker2:04323] *** End of error message ***
/usr/lib64/openmpi/lib/libmpi.so.40(+0x278158)[0x7fde99678158]
[worker1:03599] [14] /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Recv+0x268)[0x7fde994e48a8]
[worker1:03599] [15] ./mpi02[0x401265]
[worker1:03599] [16] /lib64/libc.so.6(+0x2a088)[0x7fde9923d088]
[worker1:03599] [17] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7fde9923d14b]
[worker1:03599] [18] ./mpi02[0x4010c5]
[worker1:03599] *** End of error message ***
[mce-eseiaat:05759] *** Process received signal ***
[mce-eseiaat:05759] Signal: Aborted (6)
[mce-eseiaat:05759] Signal code: (-6)
[mce-eseiaat:05759] [ 0] /lib64/libc.so.6(+0x40710)[0x7f4e7a053710]
[mce-eseiaat:05759] [ 1] /lib64/libc.so.6(+0x98144)[0x7f4e7a0ab144]
[mce-eseiaat:05759] [ 2] /lib64/libc.so.6(gsignal+0x1e)[0x7f4e7a05365e]
[mce-eseiaat:05759] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7f4e7a03b902]
[mce-eseiaat:05759] [ 4] /lib64/libfabric.so.1(+0xfc7f)[0x7f4e7960fc7f]
[mce-eseiaat:05759] [ 5] /lib64/libfabric.so.1(+0x64f9b2)[0x7f4e79c4f9b2]
[mce-eseiaat:05759] [ 6] /lib64/libfabric.so.1(+0x674d57)[0x7f4e79c74d57]
[mce-eseiaat:05759] [ 7] /lib64/libfabric.so.1(+0x6770e1)[0x7f4e79c770e1]
[mce-eseiaat:05759] [ 8] /lib64/libfabric.so.1(+0x68d2c1)[0x7f4e79c8d2c1]
[mce-eseiaat:05759] [ 9] /lib64/libfabric.so.1(+0x61ff1e)[0x7f4e79c1ff1e]
[mce-eseiaat:05759] [10] /lib64/libfabric.so.1(+0x60f303)[0x7f4e79c0f303]
[mce-eseiaat:05759] [11] /lib64/libfabric.so.1(+0x62bcc9)[0x7f4e79c2bcc9]
[mce-eseiaat:05759] [12] /usr/lib64/openmpi/lib/libmpi.so.40(+0x1d6e9b)[0x7f4e7a3d6e9b]
[mce-eseiaat:05759] [13] /usr/lib64/openmpi/lib/libmpi.so.40(+0x27990a)[0x7f4e7a47990a]
[mce-eseiaat:05759] [14] /usr/lib64/openmpi/lib/libmpi.so.40(MPI_Send+0x235)[0x7f4e7a2e52c5]
[mce-eseiaat:05759] [15] ./mpi02[0x401227]
[mce-eseiaat:05759] [16] /lib64/libc.so.6(+0x2a088)[0x7f4e7a03d088]
[mce-eseiaat:05759] [17] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7f4e7a03d14b]
[mce-eseiaat:05759] [18] ./mpi02[0x4010c5]
[mce-eseiaat:05759] *** End of error message ***
With Fedora 39, the same program worked fine. Does anybody knows why I can
not run an openmpi program with hosts os different networks?
Thanks in advance,
Rafel Amer
—
Reply to this email directly, view it on GitHub
<#12519>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAE45DKT36AIPR5MOJBYFQLZAPNKHAVCNFSM6AAAAABHF753TCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TQMRVGQZDMOA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
from ompi.
rafelamer
commented on June 2, 2024
I have tried
mpirun --mca btl_tcp_if_include 195.201.223.246/32,162.55.213.49/32,88.198.157.233/32 -np 16 --hostfile ~/hosts ./mpi02
but I give the same error
Best regards,
Rafel
from ompi.
wenduwan
commented on June 2, 2024
Note that the backtrace also shows libfabric - what is your network type?
@rafelamer Could you fill out the standard issue template from here?
from ompi.
bosilca
commented on June 2, 2024
What you did allowed for all interfaces, so it is normal you endup with the same error. Try a more restrictive appraoch, using a single network where all nodes can be reached.
Try this mpirun --mca btl_tcp_if_include 195.201.223.246/32 -np 16 --hostfile ~/hosts ./mpi02.
from ompi.
wenduwan
commented on June 2, 2024
Duplicated of #12520. Closing.
from ompi.
Related Issues (20)
- Open MPI fails with 480 processes on a single node HOT 5
- configure: error: C compiler cannot create executables HOT 6
- Unable to run openMPI from two machines HOT 5
- mca_pml_ob1_recv_frag_callback_match occasional segfault HOT 9
- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. HOT 5
- OpenMPI configure script wrongly recognizes which directive to use for ignoring tkr in case of the new LLVM Fortran compiler HOT 9
- mpirun 5.0.2 hangs - ssh works HOT 11
- --with-cuda failes to find libcuda.so HOT 4
- Scaling issue run openmp on a cluster HOT 4
- openmpi osc_ucx_component error HOT 4
- Errors when running mpi programs HOT 5
- Trying to run MPI 3.0.6 on docker HOT 6
- problem with MPI_Comm_Create_Group HOT 8
- Error `Could not find viable pmix build` while building in Docker HOT 2
- COLL/UCC doesn't compile against head of UCC at master HOT 2
- Support zero-copy non-contiguous send HOT 4
- OpenMPI/5.0.3 with PMIx/4.2.7 compilation error HOT 2
- Configure --with-tm=/opt/pbs/ with PBS Professional fails with openmpi-5.0.3, succeeds with openmpi-4.1.4 HOT 8
- Failed to build RPM from SRPM because of large UID and old tar command HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ompi.