#Apache Hadoop 2.7.1 Docker image

Hadoop in Docker with Networking and Persistent Storage

The Portworx docker-hadoop project is a collection of Docker Images for running a multi-node Hadoop cluster. Whereas other Hadoop Docker Images are meant to be run in a single image, or one host, these images can be used in a multi-host environment with Docker 1.9 networking.

Currently the only Hadoop version supported is 2.7.1

The Images also support Portworx Persistent Storage layer for Docker. With Persistent Storage Volumes the Namenode and all Datanodes can be stopped, restarted, migrated and cloned while retaining all data.

Volumes are currently configured for /hdfs/volume1

• portworx/namenode

  The namenode and secondary-namenode services run on this image. At startup it will configure itself with the hostname of the instance.

• portworx/yarn

  The Resource Manager and Job History Server run on this image. Requires the HADOOP_HOST_NAMENODE environment variable to be set to the hostname of the Namenode instance at startup. Otherwise will set the short hostname to "namenode".

• portworx/datanode

  The Datanode and NodeManager services run on this image. At startup two environment variables are required: 1) HADOOP_HOST_NAMENODE environment variable to be set to the hostname of the Namenode; HADOOP_HOST_YARN variable set to the hostname of the Yarn (Resource Manager) instance.

These images are all available from the Docker Hub automated build repository.

docker pull portworx/namenode:2.7.1

docker pull portworx/yarn:2.7.1

docker pull portworx/datanode:2.7.1

Make sure you have pulled all three images. Also make sure that SELinux is disabled on the host.

All of the images come bundled with the same set of SSH keys for user equivalence. Note that this setup isn't meant to be run in secure environments.

To create a cluster with a Namenode, Yarn server, and 3 Datanode’s issue the following docker run commands.

docker network create hadoopnet

docker run -itd -p 50070:50070 --net=hadoopnet --name namenode portworx/namenode

docker run -itd -p 8088:8088 -p 19888:19888 --net=hadoopnet --name hadoop-yarn -e HADOOP_HOST_NAMENODE=[namenode] portworx/yarn

docker run -itd --net=hadoopnet --name hadoop-node1 -e HADOOP_HOST_NAMENODE=[namenode] -e HADOOP_HOST_YARN=[yarn] portworx/datanode

docker run -itd --net=hadoopnet --name hadoop-node2 -e HADOOP_HOST_NAMENODE=[namenode] -e HADOOP_HOST_YARN=[yarn] portworx/datanode

docker run -itd --net=hadoopnet --name hadoop-node3 -e HADOOP_HOST_NAMENODE=[namenode] -e HADOOP_HOST_YARN=[yarn] portworx/datanode

Namenode - http://[namenode]:50070

Resource Manager - http://[yarn]:8088

Job History - http://[yarn]:19888

#Running Hadoop examples:

docker exec -it [yarn instance] /bin/bash

Set PATH=$PATH:/usr/local/hadoop/bin HADOOP_PREFIX=/usr/local/hadoop

cd $HADOOP_PREFIX
hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar teragen 10000 /teragen

hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar randomwriter randomout
# Check output
bin/hdfs dfs -cat randomout/*

Much of this work is based on the sequenceiq/hadoop-docker image.

If you have questions or would like to contribute to this project send an email to [email protected]