storm on yarn application master

This is to provide basic storm on YARN functionality. It is still a work in progress.

• Andy Feng (@anfeng)
• Robert Evans (@revans2)
• Derek Dagit (@d2r)
• Nathan Roberts (@ynroberts)

To run the tests you need an instance of storm installed and reachable on your PATH.

mvn package

If you want to skip the tests you can run

mvn package -DskipTests

You need to install a version of storm on the hadoop gateway.

You also need to place a corresponding storm.zip file in HDFS so it can be shipped to all of the nodes through the distributed cache at

/lib/storm/<storm-version>/storm.zip

If you are running storm using zeromq and jzmq you need to augment the standard storm.zip file the needed .so files. This can be done with the not ideally named create-tarball.sh script

create-tarball.sh storm.zip

Ideally the storm.zip file is a world readable file installed by ops so there is only one copy in the distributed cache ever.

The yarn-storm command provides a way to launch a storm cluster. In the future it is intended to also provide ways to manage the cluster.

To launch a cluster you can run

storm-yarn launch [<storm-yarn config>]

storm-yarn configuration will be used to launch a Storm cluster. storm-yarn has a number of new config options to configure the storm AM.

• master.initial-num-supervisors is the number of supervisors to launch with storm.
• master.container.size-mb is the size of the container to request.

For a full list of commands and options you can run

storm-yarn help

The is no failover when nimbus goes down. Still working on it.

There is no easy way to get the normal storm client to connect to the nimbus server that was just launched. (You need to find the host and port and then configure it automatically on the command line.)

The default UI port is the same as the default MR shuffle port. You probably want to change that to avoid collisions.

There is no simple way to get to the logs for the different processes.

The use and distribution terms for this software are covered by the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.html).