Configuring a Hadoop Cluster with Raspberry Pi's
To be clear, the following setup is impractical and should be done purely as a learning excercise. The JVM (Java Virtual Machine) which powers Hadoop is horridly slow on the ARM CPUs that drive the Raspberry Pi. Feel free to use any model Raspberry Pi as none of them will be able to do any timely data crunching anyways. This project exists solely to help in booting up your first Hadoop Environment!
Technically you could draw more computing power out of the setup with a few of the following considerations, but that's beyond the scope of this guide: Hard Float OS, Oracle JDK, JVM tuning params.
The following items are not all necessary. Obviously, you could have any combination of equipment. Technically you could run all of the hadoop tools on a single node if you wanted. By choosing these items we're just laying out the most common path we assume people will want to take. If anything below is specific to these items, we'll point it out.
Requirements:
- 3x Raspberry Pi Model B
- 3x 32GB SD Card
- 3x MicroUSB Power Cable + Charger
- 4x Ethernet Cable
- 1x 5 port unmanaged switch
- 1x power strip
Rack Setup:
power
|
----- 1 power strip
| | |
[][][] 3 nodes
| | |
----- 1 network switch
|
router
Raspberry Pi OS installation methods change regularly. At the time of writing this guide, NOOBS was the ease-of-use installer and so that's what we'll describe here.
- Download NOOBS Raspbian Installer
- Format SD Card - FAT32 Single Partition
- Unzip NOOBS zip file
- Transfer contents to SD Card
Hook up the Raspberry Pi to your TV & keyboard, boot into Noobs
- Install Raspbian
- Reboot into raspi-config
- Change default password
- Disable boot to GUI
- Fix localization options: en-US UTF8, US Central Timezone
- Enable SSH server
- Reallocate GPU memory to the smallest amount (16)
- Overclock if you want to
- Setup the hostname: hc0000, hc0001, hc0002
Set up static ip's for each node. This can be done through your router's web panel. Each node is filtered to an IP by it's ethernet port's mac address. For our examples, we'll use the following:
- hc0000 - 192.168.1.10
- hc0001 - 192.168.1.11
- hc0002 - 192.168.1.12
Most of the steps from here forward assume you will be administering your cluster from a Linux command line.
Let's setup passwordless ssh from your computer to all 3 nodes. We'll need to copy the public key from your computer out to the cluster:
cat .ssh/id_rsa.pub
ssh pi@hc0000
mkdir .ssh
echo "${public-key}" >> .ssh/authorized_keys
For the rest of this section we can run a bash loop to hit all 3 boxes at once. Something simple, like the following.
echo "hc0000
hc0001
hc0002" | while read line
do
echo $line
ssh -n pi@$line "uptime"
echo
done
This will run the command uptime
on all 3 nodes.
For the sake of clarity, I'll refrain from pasting the loop at each step but instead just the single commands.
Update the system
sudo apt-get update; sudo apt-get -y upgrade
Install Java
sudo apt-get -y install openjdk-7-jdk
Setup 'hduser' User and 'hadoop' Group
sudo addgroup hadoop
sudo adduser --ingroup hadoop hduser
Adding the user may fail when trying to do so over ssh. This is because you can't set the password without an interactive shell.
Here we'll give hduser
sudo access. This step must be done as root
user or someone with sudo access.
sudo adduser hduser sudo
WARNING: Everything from this point forward has to be run as user 'hduser'
Setup SSH so the nodes can talk to themselves and eachother
ssh-keygen -t rsa -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Test that it's working
ssh localhost
Download Hadoop
cd ~
wget http://apache.mirrors.tds.net/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
Install Hadoop to /usr/local
sudo tar vxzf hadoop-1.2.1.tar.gz -C /usr/local/
cd /usr/local
sudo mv hadoop-1.2.1/ hadoop
sudo chown -R hduser:hadoop /usr/local/hadoop
Configure User Environment
cd ~
rm hadoop-1.2.1.tar.gz
You can use any editor you like but I'll denote the moments when we're supposed to edit a text file with vim.
vim .bash_profile
Add the following to each node's ~/.bash_profile
# Setup Environment Variables
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
Setup the hadoop-env.sh
vim /usr/local/hadoop/conf/hadoop-env.sh
The options to adjust here are:
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf
export HADOOP_HEAPSIZE=384
Some other similar guides had set HADOOP_HEAPSIZE
to 272. I raised it up a bit because there was 433 MB free upon issuing a free -m
after cold boot.
Setup the tmp dir and filesystem URI
vim /usr/local/hadoop/conf/core-site.xml
Add the following to the section:
<property>
<name>hadoop.tmp.dir</name>
<value>/fs/hadoop/tmp</value>
<description>Sets the operating directory for Hadoop data.
</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation.
The URI's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The URI's authority is used to
determine the host, port, etc. for a filesystem.
</description>
</property>
Set up the job tracker URI
vim /usr/local/hadoop/conf/mapred-site.xml
Add the following to the section:
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
Set the replication factor to 1 for now, since we only have 1 node.
vim /usr/local/hadoop/conf/hdfs-site.xml
Add the following to the section:
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
Configure the hdfs on real disk
sudo mkdir -p /fs/hadoop/tmp
sudo chown hduser:hadoop /fs/hadoop/tmp
sudo chmod 750 /fs/hadoop/tmp/
Format the HDFS
/usr/local/hadoop/bin/hadoop namenode -format
Start it up!
/usr/local/hadoop/bin/start-all.sh
You can run 'jps' command to ensure all the appropriate Java Hadoop processes are running:
DataNode
TaskTracker
SecondaryNameNode
NameNode
JobTracker
If you've encountered failures look at the logs:
/usr/local/hadoop/logs
Logs are titled like:
hadoop-(username)-(processtype)-(machinename).log
If everything is working as expected you should be able to hit the following pages:
- NameNode - 192.168.1.10:50070
- JobTracker - 192.168.1.10:50030
- TaskTracker - 192.168.1.10:50060
Now that everything is up and running we'll run our first job, a word count example!
Download any book in "Plain Text UTF-8" format from www.gutenburg.org and load it into HDFS:
mkdir -p /tmp/books
cd /tmp/books
wget http://www.gutenberg.org/files/43790/43790-0.txt
mv 43790-0.txt bookofcats.txt
/usr/local/hadoop/bin/hadoop dfs -copyFromLocal /tmp/books /fs/hduser/books
Run the wordcount example:
/usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/hadoop*examples*.jar wordcount /fs/hduser/books /fs/hduser/books-output
Now, let's work with multiple nodes.
Master node, hc0000, will run:
NameNode
Secondary NameNode
DataNode
JobTracker (In a large production cluster this would be somewhere else)
TaskTracker
In a large production cluster Secondary NameNode and JobTracker would have their own servers.
Slave nodes, hc0001 and hc0002, will run:
DataNode
TaskTracker
Before we begin again, setup passwordless ssh between all nodes:
Gather up all keys with a loop
cat ~/.ssh/id_rsa.pub
Take these public keys and add them to the authorized_keys file of each node with a loop
echo "${public_keys}" >> ~/.ssh/authorized_keys
These two files, masters and slaves, control the way the cluster stops and starts. You can remove hosts from here and reboot the cluster to kick a node out, or conversely bring one in.
vim /usr/local/hadoop/conf/masters
Change:
localhost
To:
hc0000
vim /usr/local/hadoop/conf/slaves
Change:
localhost
To:
hc0000
hc0001
hc0002
vim /usr/local/hadoop/conf/core-site.xml
Change:
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
To reflect the hostname of the server running NameNode:
<name>fs.default.name</name>
<value>hdfs://hc0000:54310</value>
vim /usr/local/hadoop/conf/mapred-site.xml
Change:
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
To:
<name>mapred.job.tracker</name>
<value>hc0000:54311</value>
vim /usr/local/hadoop/conf/hdfs-site.xml
Change:
<name>dfs.replication</name>
<value>1</value>
To:
<name>dfs.replication</name>
<value>3</value>
Copy core-site.xml, mapred-site.xml and hdfs-site.xml out from the master to the new nodes:
scp hduser@hc0000:/usr/local/hadoop/conf/core-site.xml hduser@hc0001:/usr/local/hadoop/conf/.
scp hduser@hc0000:/usr/local/hadoop/conf/mapred-site.xml hduser@hc0001:/usr/local/hadoop/conf/.
scp hduser@hc0000:/usr/local/hadoop/conf/hdfs-site.xml hduser@hc0001:/usr/local/hadoop/conf/.
scp hduser@hc0000:/usr/local/hadoop/conf/core-site.xml hduser@hc0002:/usr/local/hadoop/conf/.
scp hduser@hc0000:/usr/local/hadoop/conf/mapred-site.xml hduser@hc0002:/usr/local/hadoop/conf/.
scp hduser@hc0000:/usr/local/hadoop/conf/hdfs-site.xml hduser@hc0002:/usr/local/hadoop/conf/.
Configure the hdfs on real disk
sudo mkdir -p /fs/hadoop/tmp
sudo chown hduser:hadoop /fs/hadoop/tmp
sudo chmod 750 /fs/hadoop/tmp/
sudo vi /etc/hosts
Change:
127.0.1.1 hc0000
To:
192.168.1.10 hc0000
192.168.1.11 hc0001
192.168.1.12 hc0002
Leave the rest of the file intact.
Reboot each node to make sure these settings update
sudo reboot
Format the HDFS
/usr/local/hadoop/bin/hadoop namenode -format
Start it up
/usr/local/hadoop/bin/start-all.sh
You can run 'jps' again to make sure all the right processes are up and running.
You should now be able to view the following web pages:
NameNode
192.168.1.10:50070
JobTracker
192.168.1.10:50030
TaskTrackers
192.168.1.10:50060
192.168.1.11:50060
192.168.1.12:50060
Congratulations on setting up your first Hadoop Environment!
A great resource for understanding Hadoop setups: http://bradhedlund.com/2011/09/10/understanding-hadoop-clusters-and-the-network/
This tutorial is mostly based on the efforts of the following blog post: http://blog.ittoby.com/2013/08/starting-small-set-up-hadoop-compute.html