ofiwg / librdmacm Goto Github PK

If this isn't the right place to report this, then please redirect or forward.

We've seen processes that use the rsocket library hang indefinitely.
Here's a backtrace from such a process (obtained via gdb -p ...)

#0  0x00007f7d76bdf3ad in read () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f7d73e611f3 in read (__nbytes=8, __buf=0x7fff8c56c850, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2  __ibv_get_cq_event (channel=<optimized out>, cq=0x7fff8c56c880, cq_context=0x7fff8c56c888) at src/verbs.c:484
#3  0x00007f7d75688394 in ?? () from /usr/lib/librdmacm.so.1
#4  0x00007f7d75689ec0 in ?? () from /usr/lib/librdmacm.so.1
#5  0x00007f7d7568d23e in rshutdown () from /usr/lib/librdmacm.so.1
#6  0x00007f7d7568ec1a in rclose () from /usr/lib/librdmacm.so.1
#7  0x0000000000ddf454 in ordma_rclose (fd=<optimized out>) at ordma_close.c:28

This is version 1.0.21 (installed via debian package: librdmacm1 1.0.21mlnx-OFED.3.3.0.0.1.33100 )

It seems to be related to this SO question:
http://stackoverflow.com/questions/38119745/how-to-avoid-getting-stuck-in-rdma-get-recv-comp-or-ibv-get-cq-event

In my case, the memory comes from mmap and during mmap, the prot argument is PROT_READ and rdma_reg_msgs always report an error. I think the reason is that rdma_reg_msgs requires the memory can be write and read, but the mmap don't give the write permission.
So is there any possible that mmap only gives read permission and some register API in librdmacm support register the readonly memory?

Hi,
Is there a plan to update rsocket in the future? rsocket now has rselect,rpoll, so epoll-like interfaces will be added?

Thanks

Hi,
We are using librdmacm.so.1.1.16.9, which provides RDMA APIs (e.g., rdma_create_ep, rdma_listen, rdma_connect) for user-space apps.

However, for a single user-space process using this lib, we found that only 339 QPs can be created.
So could you help to confirm if there is a limitation on the number of QPs that a single user-space process can create using this lib?

Following is some of the logs from our test (performed on a VM with the rdma_rxe driver):

System info:
linux-15v9:~ # cat /etc/os-release
NAME="SLES"
VERSION="12-SP4"
VERSION_ID="12.4"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp4"

linux-15v9:~ # uname -r
4.12.14-94.41-default

linux-15v9:~ # lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 142
Model name: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz
Stepping: 12
CPU MHz: 2112.002
BogoMIPS: 4224.00
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xsaves arat flush_l1d arch_capabilities

Server side:
linux-15v9:~ # ib_perf.exe --server-ip 192.168.17.150 --server-port 10001 -s --qp-num 1024
qp [0] local 192.168.17.150:10001 peer 192.168.17.150:51705 created.
qp [1] local 192.168.17.150:10001 peer 192.168.17.150:37190 created.
......
qp [337] local 192.168.17.150:10001 peer 192.168.17.150:43572 created.
qp [338] local 192.168.17.150:10001 peer 192.168.17.150:52580 created.

Client side:
linux-15v9:/usr/lib64 # ib_perf.exe --server-ip 192.168.17.150 --server-port 10001 -c --qp-num 1024
qp [0] local 192.168.17.150:51705 peer 192.168.17.150:10001 created.
qp [1] local 192.168.17.150:37190 peer 192.168.17.150:10001 created.
......
qp [337] local 192.168.17.150:43572 peer 192.168.17.150:10001 created.
qp [338] local 192.168.17.150:52580 peer 192.168.17.150:10001 created.
ERR_DBG:/mnt/linux-dev-framework-master/apps/ib_perf/perf_frmwk.c(599)-create_connections_client:
rdma_create_ep failed: Cannot allocate memory

Hello,
Here is the quote from man for poll():

Specifying a negative value in timeout means an infinite timeout.

But it seems that it doesn't work for rpoll(). Is it a bug? Can you advise a workaround?

Im trying to install this but get the following error -

(tf_basic) pr@test-compile:~/librdmacm$ ./autogen.sh

• test -d ./config
• autoreconf -ifv -I config
  ./autogen.sh: 5: ./autogen.sh: autoreconf: not found

From the man page of the select() function, we can use the function as a subsecond precision timer. However, the rselect() does not support it and it seems it just returns an ENOMEM error. Is there any plan to support this?

Here is an extract from the man page.
Some code calls select() with all three sets empty, nfds zero, and a non-NULL timeout as a
fairly portable way to sleep with subsecond precision.

Hello,

I have been experimenting lately with rdmacm(1.0.17) library and something strange hapenned.
I am having a single passive side(server) and issuing reads(of 128 MB) from a client to a remote memory. The latency to receive completions for multiple buffers is ok when posting multiple reads. But when adding another queue pair (even if this queue pair have to do with 2 different nic's/2 different physical machines) The latency becomes very high sometimes for some completion events. For a single read request of 128 MB rdma_get_send_comp can take 1 2 or 3 seconds which I guess is not normal.
Some completion events have small delay but at some point the delay becomes orders of magnitude higher.
Is there something I am missing here? Or anything that I might be doing wrong?