atcp / hotpot Goto Github PK

Hotpot is a kernel-level, RDMA-based Distributed Shared Persistent Memory (DSPM) system. Applications can access data through memory load/store instructions and at the same time make the data durable and survive various types of failures.

We built Hotpot as a linux module for the Linux 3.11.1 kernel (a small part of the original kernel is changed because of a limitation of the 3.11.1 kernel and will not be necessary for newer kernels). The Hotpot kernel module is in hotpot/. The folder hotpot/test has some simple examples of using Hotpot.

This is a BETA version, use under your own risk! We will have our stable version ready soon.

For more information, check https://engineering.purdue.edu/WukLab/hotpot-socc17.pdf

1. More than two machines connected via InfiniBand.
2. One of the machines (served as central dispatcher) has installed InfiniBand OFED user-level library. The rest of the machines serve as Hotpot nodes and need to install hotpot-kernel (see below).

Hotpot's CD source code is located in hotpot/server/, which runs on user space. Assume this machine has installed all IB user libraries, you can go to this directory and simply do make. After that, you will have a hotpot-server, which is our CD server. Also, get the IP address of this CD server, which will be used by all other hotpot nodes to establish connection.

1. First, compile the hotpot-kernel using hotpot-kernel directory. Compile the kernel with your machine's old config:
   cp /boot/config-your-default-kernel-version hotpot-kernel/.config
make oldconfig (Recommended to have a special CONFIG_LOCALVERSION="-hotpot")
   make && make modules && make modules_install && make install

2. To run Hotpot, a contiguous physical memory region must be reserved for Hotpot usage. To do so, CentOS users could open /boot/grub2/grub.cfg and find hotpot-kernel's entry. Append memmap=N[KMG]\$S[KMG] to kernel parameter. The actual parameter depends on your usage. For example, to reserve [4G - 20G], you can append memmap=16G\$4G.

3. Reboot the machine and 1) use uname to check if the kernel version matches. 2) Use dmesg or free to check if memory has been reserved. x86 users can also check e820 tables.

Hotpot has several options that can be configured at compile time. The default configurations have been tested to work well for our applications. We will provide a documentation of these configurations soon.

After boot into hotpot-kernel successfully (S2), go to hotpot directory and type make to compile two modules. If the kernel is right, you will have 2 modules compiled: hotpot.ko and hotpot_net.ko. hotpot.ko is the Hotpot module, hotpot_net.ko is a customized RDMA-stack which Hopot runs on top of.

In general, to run hotpot, you need to start CD server first, which will listen on a port you specified. After that, start hotpot node one by one to establish the connection with CD server.

Assume the IP address of CD is 192.168.1.1, and you want CD to listen on port 18500, then you can start CD server like this:

./hotpot-server -l 18500

There is a simple script hotpot/run.sh, which help us to install modules and mount hotpot's filesystem. The hotpot's filesystem interface is used to simplify our programming experience by supporting commonly used POSIX APIs, e.g., open, close, and msync. After you run ./run.sh 1, you should be able to see some output at CD side. To connect multiple Hotpot nodes, just do the above steps one by one.

In detail:

1. insmod hotpot_net.ko ip=192.168.1.1 port=18500
   This will insmod hotpot network module
   ip=192.168.1.1 port=18500 need to match CD's setting
2. insmod hotpot.ko
3. mount -t hotpot -o /mnt/hotpot none physaddr=4G,size=4G,verbose,dbgmask=0
   Mount /mnt/hotpot
   [physaddr, physaddr+size) must fully fall into memmap reserved area.
   (Check run.sh for more details)

There are several code samples under hotpot/test/. Basically, we open (or create) a dataset by calling POSIX open. After that, the opened fd will be mmap'ed into application's address space. If mmap succeed, application can access the DSPM space directly and transpatently.

Hotpot will create two special files: /proc/dsnvm-event and /proc/dsnvm. The first one lists a lot of hotpot internal activities, which will help us to understand what is going within the system. The latter one lists some general informations. Both of them will help us debug and tune the system. If you have any issues with deploying Hotpot, please contact Yizhou Shan [email protected].

@inproceedings{Shan17-SOCC-Hotpot,
author = {Yizhou Shan and Shin-Yeh Tsai and Yiying Zhang},
title = {Distributed Shared Persistent Memory},
booktitle = {Proceedings of the 8th Annual Symposium on Cloud Computing (SOCC '17)},
year = {2017},
address = {Santa Clara, CA, USA},
month = {September}
}