I'm writing some IBStream like below

pub struct IBStream<'a> {
    qp: Arc<ibverbs::QueuePair<'a>>,
    cq: Arc<ibverbs::CompletionQueue<'a>>,
    mr: ibverbs::MemoryRegion<RdmaPrimitive>,
    pd: Arc<ibverbs::ProtectionDomain<'a>>,
    ctx: Arc<ibverbs::Context>,
}

and if I want to write the new func:

    pub fn new<'b, A: ToSocketAddrs>(addr: A) -> Result<IBStream<'b>, IBError> {
        let ctx = Self::setup_ctx()?;
        let ctxr: & _ = &ctx;
        let cq = Self::setup_cq(ctxr)?;
        let pd = Self::setup_pd(&ctx.clone())?;
        let qp = Self::setup_qp(&cq.clone(), &pd.clone(), &ctx.clone())?;

        let mr = pd.allocate::<RdmaPrimitive>(2).unwrap();
        Ok(IBStream {
            qp,
            cq,
            mr,
            pd,
            ctx,
        })
    }

The lifetime of ctx is in accordance with the whole struct and could not specify any lifetime hint to make it happen. But if I roll back to this version and remove the lifetime, the code compiled nice and sound. If I have to keep using the latest version, I have to take the ctx out of IBStream.

I really appreciate this library but there is one thing which concerns me: integration testing.

I'm looking for feedback on the desired approach here (or alternatives if you think of any).

To be clear, I don't really feel that unit testing is especially problematic in this case: bindgen takes care of
validating the bindings and upstream rdma-core is already well tested in its own right. That said, there are a number of really handy convenience methods and data structures which I would like to help test.

I iterated through several options for integration testing here and I have basically concluded "it is going to be hard."
Still, that doesn't mean I shouldn't try.

Options

1. Vagrant virtual machines + SoftRoCE: workable but a bit heavy.
   This will require significant tooling work and is likely to be really problematic if ya want to continue to use a
   tool like TravisCI.

2. Docker container + SoftRoCE: workable, not super portable, challenging for CI.
   In the name of experimentation, I cooked up a debian-bullseye container which builds the project, sets up a
   SoftRoCE interface, and runs a few integration tests. The problem is that RDMA is, at its heart, a hardware offload.
   Even with SoftRoCE you are just emulating a hardware offload. As long as you are using the host's Kernel (as would
   be the case for most container technologies) then the container is really only buying you a consistent development
   environment; not a consistent deployment environment. For example, if the CI system hasn't loaded the SoftRoCE
   module (rxe) then the container won't work anyway. Packaging the SoftRoCE kernel module with the container is
   useless due to ABI compatibility problems with differing kernel versions. You might be able to somehow overlay mount
   /lib/modules/ from the host and use DKMS to build and load the SoftRoCE module on a per kernel basis but this seems
   like a huge pain. At the very least this will be very difficult to make reliable and portable.

   In my experience the best way to ensure tests never get written is to make them a huge pain to write and maintain.

   Oh, one more problem: RDMA devices are not (by default) network namespaced. Even when I enable RDMA namespacing, I
   can't seem to get SoftRoCE devices to move namespaces. Thus my dream of setting up something like mininet for RoCE
   doesn't seem viable currently.

3. Bash script sets up SoftRoCE device, assume developer has taken care of the kernel config: not ideal but I think it
   is the best available choice at the moment.

   I have already composed a simple bash script which sets up a single SoftRoCE device. This isn't really portable
   either and it requires that the developer already have rdma-core installed (it requires iproute2 and the rdma
   command). I don't know how CI would react to this configuration.

Thoughts?

Hello,

I'm trying out the library for a project, for now doing some tests using Soft-RoCE on a pair of Ubuntu20.10 VMs.

I cloned the repo and ran the example, and I inconsistently encounter the following error:
panicked at 'called Result::unwrap() on an Err value: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }', ibverbs/examples/loopback.rs:16:10.

The code that gives the issue is
let qp_builder = pd.create_qp(&cq, &cq, ibverbs::ibv_qp_type::IBV_QPT_RC).build().unwrap();,
in particular the call in .build() to ibv_create_qp. I've run it through the debugger and have not been able to spot any differences between successful and unsuccessful runs.

I would say it happens about 50% of the time, and for the rest the code runs fine. Even more strangely, I tried moving the Protection Domain memory allocation code above the queue pair builder and found that it fixed the issue, but I have no idea why:

let mut mr = pd.allocate::<u64>(2).unwrap();
mr[1] = 0x42;

let qp_builder = pd.create_qp(&cq, &cq, ibverbs::ibv_qp_type::IBV_QPT_RC).build().unwrap();

System details:

I'm using this Vagrant VM image and the rdma_rxe driver for Soft-RoCE, which I load using sudo modload rdma_rxe before adding a soft-roce device: sudo rdma link add rxe_0 type rxe netdev eth1. I've tested the setup using ib_send_bw, which works.

Any idea what could be causing the problem?

I came across your bindings and decided to give them a whirl - I was almost going to create my own!

However, I can't get them to build. I'm using Mac OS X, with a local linux / musl toolchain (installed via brew install filosottile/musl-cross/musl-cross). Its works well, and I used it to cross-build nvml, bearssl, DPDK, libfabric and others... however, the build.rs doesn't seem to respect --target from cargo, and so consequently fails. (I'm not entirely sure that's the case, though).

I realise it's early days for this binding, so it's probably something that's not on the radar yet, and might be something you don't want to cope with.

Hello,

when I try to compile rust-ibverbs, I get an error with the following error message during compilation of the rdma-core package.

Scanning dependencies of target sminfo
Scanning dependencies of target saquery
Scanning dependencies of target ibtracert
Scanning dependencies of target ibcacheedit
Scanning dependencies of target testleaks
In file included from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/build/include/ccan/minmax.h:7,
                 from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/librdmacm/cma.h:48,
                 from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/librdmacm/acm.c:43:
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/librdmacm/acm.c: In function 'ucma_ib_init':
Scanning dependencies of target perfquery
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/build/include/ccan/build_assert.h:23:26: error: size of unnamed array is negative
   23 |  do { (void) sizeof(char [1 - 2*!(cond)]); } while(0)
      |                          ^
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/librdmacm/acm.c:105:3: note: in expansion of macro 'BUILD_ASSERT'
  105 |   BUILD_ASSERT(sizeof(IBACM_SERVER_PATH) <=
      |   ^~~~~~~~~~~~
Scanning dependencies of target ibsysstat
Scanning dependencies of target ibccquery
Scanning dependencies of target ibqueryerrors
Scanning dependencies of target dump_fts
Scanning dependencies of target smpquery
Scanning dependencies of target ibportstate
Scanning dependencies of target ibstat
Scanning dependencies of target smpdump
Scanning dependencies of target ibnetdiscover
Scanning dependencies of target ibping
Scanning dependencies of target ibccconfig
Scanning dependencies of target ibroute
Scanning dependencies of target iblinkinfo
Scanning dependencies of target ibaddr
In file included from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/build/include/ccan/minmax.h:7,
                 from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/linux/osd.h:48,
                 from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/src/acm.c:38:
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/src/acm.c: In function 'acm_listen':
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/build/include/ccan/build_assert.h:23:26: error: size of unnamed array is negative
   23 |  do { (void) sizeof(char [1 - 2*!(cond)]); } while(0)
      |                          ^
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/src/acm.c:629:3: note: in expansion of macro 'BUILD_ASSERT'
  629 |   BUILD_ASSERT(sizeof(IBACM_IBACME_SERVER_PATH) <=
      |   ^~~~~~~~~~~~
In file included from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/build/include/ccan/minmax.h:7,
                 from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/linux/osd.h:48,
                 from /home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/src/libacm.c:33:
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/src/libacm.c: In function 'ib_acm_connect_unix':
Scanning dependencies of target vendstat
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/build/include/ccan/build_assert.h:23:26: error: size of unnamed array is negative
   23 |  do { (void) sizeof(char [1 - 2*!(cond)]); } while(0)
      |                          ^
/home/mplaneta/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/ibacm/src/libacm.c:107:3: note: in expansion of macro 'BUILD_ASSERT'
  107 |   BUILD_ASSERT(sizeof(IBACM_IBACME_SERVER_PATH) <=
      |   ^~~~~~~~~~~~
make[2]: *** [librdmacm/CMakeFiles/rdmacm.dir/build.make:82: librdmacm/CMakeFiles/rdmacm.dir/acm.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2512: librdmacm/CMakeFiles/rdmacm.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: *** [ibacm/CMakeFiles/ib_acme.dir/build.make:95: ibacm/CMakeFiles/ib_acme.dir/src/libacm.c.o] Error 1

The issue happens it this line:

  BUILD_ASSERT(sizeof(IBACM_SERVER_PATH) <=
                             sizeof(addr.unx.sun_path));

inside ~/.cargo/registry/src/github.com-1ecc6299db9ec823/ibverbs-sys-0.1.0/vendor/rdma-core/librdmacm/acm.c

The reason is that IBACM_SERVER_PATH generated during configure step is too long (>108 chars). And it is too long, because it is
<build directory>/var/run/ibacm.sock.

// on host 1
let pqp: PreparedQueuePair = ...;
let host1end = pqp.endpoint();
host2.send(host1end);
let host2end = host2.recv();
let qp = pqp.handshake(host2end);

// on host 2
let pqp: PreparedQueuePair = ...;
let host2end = pqp.endpoint();
host1.send(host2end);
let host1end = host1.recv();
let qp = pqp.handshake(host1end);

According to my understanding, host 1 and host2 are bound to a process respectively. What work are done in the above, send and recv respectively, and how to use host2.send and recv on the process of host 1

Hello,

ibv_get_device_guid returns GUID in "network byte order" [1], which means big endian.

On another hand, the Rust counterpart returns u64, which I expect to be in the host byte order, which is little endian on my system. I would suggest to convert the result of the C version into host order. What would be your thoughts?

1. https://linux.die.net/man/3/ibv_get_device_guid

@jonhoo @daniel-noland wondering what plans you had for this crate. Are you planning to maintain this over time or is any company looking at sustaining the project?

A similar wrapper crate for librdmacm (RDMA connection manager) would be nice as it greatly simplifies the connection setup for small applications. Being also handled by rdma-core it fits well under the same umbrella as ibverbs.

Would it be possible to pull in a newer version of rdma-core? The commit is pinned at 3yrs ago and I think since then some new verbs have been introduced. I'm looking to get support for read+write. They're apparently faster then Receive/Send because they don't notify the remote end.

Hi Jon,

I'm trying to compile this on one of our IB machines and it looks like there's some mismatch between the bindgen bindings and ffi.rs: rustc output follows. I'd gladly help tracking this down / fixing it but some pointers (if you have any) would be extremely useful (I've never used bindgen).

Thanks!

(Oh and congrats/good luck with the submission.)

   Compiling ibverbs v0.3.0 (file:///home/andreal/Src/rust-ibverbs)
error[E0412]: cannot find type `ibv_send_wr__bindgen_ty_3` in this scope
   --> src/ffi.rs:364:18
    |
364 | impl Default for ibv_send_wr__bindgen_ty_3 {
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^ did you mean `ibv_send_wr__bindgen_ty_1`?

error[E0422]: cannot find struct, variant or union type `ibv_send_wr__bindgen_ty_3` in this scope
   --> src/ffi.rs:366:9
    |
366 |         ibv_send_wr__bindgen_ty_3 {
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^ did you mean `ibv_send_wr__bindgen_ty_1`?

error[E0063]: missing field `xrc_domain` in initializer of `ffi::ibv_qp_init_attr`
   --> src/lib.rs:697:24
    |
697 |         let mut attr = ffi::ibv_qp_init_attr {
    |                        ^^^^^^^^^^^^^^^^^^^^^ missing `xrc_domain`

error[E0560]: struct `ffi::ibv_send_wr` has no field named `qp_type`
    --> src/lib.rs:1106:13
     |
1106 |             qp_type: Default::default(),
     |             ^^^^^^^^ `ffi::ibv_send_wr` does not have this field
     |
     = note: available fields are: `wr_id`, `next`, `sg_list`, `num_sge`, `opcode` ... and 4 others

error[E0560]: struct `ffi::ibv_send_wr__bindgen_ty_2` has no field named `xrc`
   --> src/ffi.rs:358:13
    |
358 |             xrc: Default::default(),
    |             ^^^^ `ffi::ibv_send_wr__bindgen_ty_2` does not have this field
    |
    = note: available fields are: `qp_type`, `xrc_remote_srq_num`, `bindgen_union_field`

error[E0560]: struct `ffi::ibv_qp_attr` has no field named `rate_limit`
   --> src/ffi.rs:402:13
    |
402 |             rate_limit: 0,
    |             ^^^^^^^^^^^ `ffi::ibv_qp_attr` does not have this field
    |
    = note: available fields are: `qp_state`, `cur_qp_state`, `path_mtu`, `path_mig_state`, `qkey` ... and 20 others

error[E0599]: no variant named `IBV_WC_LOCAL_INV` found for type `ffi::ibv_wc_opcode` in the current scope
    --> src/ffi.rs:411:21
     |
411  |             opcode: ibv_wc_opcode::IBV_WC_LOCAL_INV,
     |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ variant not found in `ffi::ibv_wc_opcode`
     |
    ::: /home/andreal/Src/rust-ibverbs/target/debug/build/ibverbs-9097a2a478619ab2/out/bindings.rs:1566:1
     |
1566 | pub enum ibv_wc_opcode {
     | ---------------------- variant `IBV_WC_LOCAL_INV` not found here

error: aborting due to 7 previous errors

error: Could not compile `ibverbs`.

To learn more, run the command again with --verbose.

I was running into an issue related to the Drop implementation for CompletionQueue objects. I am unsure if there is a good way to handle it, or if an API improvement can be made to avoid panic. Example:

fn main() {
    let ctx = ibverbs::devices()
        .unwrap()
        .iter()
        .next()
        .expect("no rdma device available")
        .open()
        .unwrap();

    let cq = ctx.create_cq(16, 0).unwrap();
    let pd = ctx.alloc_pd().unwrap();

    let _qp_builder = pd
        .create_qp(&cq, &cq, ibverbs::ibv_qp_type::IBV_QPT_RC)
        .build()
        .unwrap();

    // something bad happens, perhaps returning Err(...)

   
}  // CompletionQueue::Drop -> panic!

A common example of "something bad happens" is an application fails to send the serialized QueuePairEndpoint. In this case, I would like the function to return gracefully and everything resume operation. However, here the Drop of CompletionQueue will panic, I believe because the queue pair is still associated with it.

Is there any existing way to make the above drop happen gracefully?

It mentions in the docs here that the memory region allocation from a protection domain has a default set of access rights:
https://github.com/jonhoo/rust-ibverbs/blob/main/ibverbs/src/lib.rs#L1173
Is it possible to modify these rights? I'm new to this crate so I apologize if I overlooked something.