ePipe is a metadata system for HopsFS that provides replicated-metadata-as-a-service. The key component of ePipe is a databus that both creates a consistent, correctly-ordered change stream from HopsFS and eventually delivers the stream with low latency (sub-second) to external stores and downstream clients.

In this setup, we compile ePipe againt different platforms (ubuntu, centos) using docker containers. Therefore, you will need to have docker installed on your machine.

Clone the ePipe repo, and then build it using docker.

git clone https://github.com/logicalclocks/ePipe.git
cd ePipe
./docker-build.sh

The docker build script will create two docker images epipe_build_ubuntu and epipe_build_centos with their tag set to be the current ePipe version. These images are created based on the corresponding docker files located in docker/ubuntu and docker/centos. Then, it will run the containers to build the ePipe code for both platforms. The resultant binaries will be found under builds directory.

Interactive mode

You can use the docker interactive mode, to build and test ePipe againt ubuntu or centos depending on the image you use. An example of running a docker container with the ubuntu image (epipe_build_ubuntu) where we mount the ePipe repo to the container. The script build.sh can then be used to build ePipe.

cd ePipe
docker run --rm -v "$PWD":/usr/epipe:z -w /usr/epipe -it epipe_build_ubuntu:0.18.0 
epipe@24d10f5aacc3:/usr/epipe$ ./build.sh

Software Required

For compiling ePipe you will need the following software.

• CMake 3.5.0 or higher (3.15.0 is recommended)
• GCC 7.0 or higher
• Boost 1.70 or higher
• RapidJson 1.1.0
• MySQL Cluster NDB

Untar the MySQL Cluster binaries in /usr/local/

cd /usr/local
wget https://dev.mysql.com/get/Downloads/MySQL-Cluster-7.6/mysql-cluster-gpl-7.6.10-linux-glibc2.12-x86_64.tar.gz
tar xzvf mysql-cluster-gpl-7.6.10-linux-glibc2.12-x86_64.tar.gz
ln -s mysql-cluster-gpl-7.6.10-linux-glibc2.12-x86_64 mysql

Clone the ePipe repo, and then build it.

git clone https://github.com/logicalclocks/ePipe.git
cd ePipe
mkdir build
cd build
cmake ..
make

To run ePipe, you need to create a config.ini file, you can copy the config.ini.template and then update the configuration accordingly, then run ePipe.

cp config.ini.template config.ini
./ePipe -c config.ini

A description of the allowed configuration parameters can be found in the config.ini.template file as well as when running the description (-d | --desc) switch on ePipe.

./ePipe -d

ePipe is installed as part of the Hopsworks platform as specified by the epipe-chef cookbook