Giter VIP home page Giter VIP logo

rdma-example's Introduction

RDMA exmaple

A simple RDMA server client example. The code contains a lot of comments. Here is the workflow that happens in the example:

Client:

  1. setup RDMA resources
  2. connect to the server
  3. receive server side buffer information via send/recv exchange
  4. do an RDMA write to the server buffer from a (first) local buffer. The content of the buffer is the string passed with the -s argument.
  5. do an RDMA read to read the content of the server buffer into a second local buffer.
  6. compare the content of the first and second buffers, and match them.
  7. disconnect

Server:

  1. setup RDMA resources
  2. wait for a client to connect
  3. allocate and pin a server buffer
  4. accept the incoming client connection
  5. send information about the local server buffer to the client
  6. wait for disconnect
How to run
git clone https://github.com/animeshtrivedi/rdma-example.git
cd ./rdma-example
cmake .
make
server
./bin/rdma_server
client
atr@atr:~/rdma-example$ ./bin/rdma_client -a 127.0.0.1 -s textstring 
Passed string is : textstring , with count 10 
Trying to connect to server at : 127.0.0.1 port: 20886 
The client is connected successfully 
---------------------------------------------------------
buffer attr, addr: 0x5629832e22c0 , len: 10 , stag : 0x1617b400 
---------------------------------------------------------
...
SUCCESS, source and destination buffers match 
Client resource clean up is complete 
atr@atr:~/rdma-example$ 

Does not have an RDMA device?

In case you do not have an RDMA device to test the code, you can setup SofitWARP software RDMA device on your Linux machine. Follow instructions here: https://github.com/animeshtrivedi/blog/blob/master/post/2019-06-26-siw.md.

rdma-example's People

Contributors

animeshtrivedi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

rdma-example's Issues

Usage of struct rdma_buffer_attr and struct ibv_mr?

Hi Mr. Trivedi,

Thank you for your contribution on rdma and your project really helps me. However, I am confused with the usage of struct rdma_buffer_attr and struct ibv_mr.

Typically, in below two parts of code. Client and Server both wants to send similar metadata(includes buffer address and key) to the other. Why sge values of each are set in different way? And why client does two times of rdma_buffer_register() while server only does one time?

In server's send_server_metadata_to_client() :

server_buffer_mr = rdma_buffer_alloc(pd /* which protection domain */, 
		       client_metadata_attr.length /* what size to allocate */, 
		       (IBV_ACCESS_LOCAL_WRITE|
		       IBV_ACCESS_REMOTE_READ|
		       IBV_ACCESS_REMOTE_WRITE) /* access permissions */);
       if(!server_buffer_mr){
	       rdma_error("Server failed to create a buffer \n");
	       /* we assume that it is due to out of memory error */
	       return -ENOMEM;
       }
       /* This buffer is used to transmit information about the above 
	* buffer to the client. So this contains the metadata about the server 
	* buffer. Hence this is called metadata buffer. Since this is already 
	* on allocated, we just register it. 
        * We need to prepare a send I/O operation that will tell the 
	* client the address of the server buffer. 
	*/
       server_metadata_attr.address = (uint64_t) server_buffer_mr->addr;
       server_metadata_attr.length = (uint32_t) server_buffer_mr->length;
       server_metadata_attr.stag.local_stag = (uint32_t) server_buffer_mr->lkey;

In client's client_xchange_metadata_with_server():


client_src_mr = rdma_buffer_register(pd,
			src,
			strlen(src),
			(IBV_ACCESS_LOCAL_WRITE|
			 IBV_ACCESS_REMOTE_READ|
			 IBV_ACCESS_REMOTE_WRITE));
	if(!client_src_mr){
		rdma_error("Failed to register the first buffer, ret = %d \n", ret);
		return ret;
	}
	/* we prepare metadata for the first buffer */
	client_metadata_attr.address = (uint64_t) client_src_mr->addr; 
	client_metadata_attr.length = client_src_mr->length; 
	client_metadata_attr.stag.local_stag = client_src_mr->lkey;
	/* now we register the metadata memory */
	client_metadata_mr = rdma_buffer_register(pd,
			&client_metadata_attr,
			sizeof(client_metadata_attr),
			IBV_ACCESS_LOCAL_WRITE);
	if(!client_metadata_mr) {
		rdma_error("Failed to register the client metadata buffer, ret = %d \n", ret);
		return ret;
	}
	/* now we fill up SGE */
	client_send_sge.addr = (uint64_t) client_metadata_mr->addr; 
	client_send_sge.length = (uint32_t) client_metadata_mr->length;
	client_send_sge.lkey = client_metadata_mr->lkey;

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.