Giter VIP home page Giter VIP logo

hadoop-build-env's Introduction

Hadoop-build-env

Automatical deployment, configuration, benchmark execution, and performance report

Here mainly focus on hadoop-3.0.0-beta1. And we take delight in contribution in efficient cluster scheduling and approaches that can help it.

How to run

  1. [optional]. Customize setting.yaml according to user demands.

  2. [optional]. Create soft/hard link in the folder within $PATH.

# sudo ln -s <absolute path>/hbe <folder in $PATH>/<name you liked>
#
# example. Enter hadoop-build-env folder 
$ sudo ln -s `pwd`/hbe /usr/bin/hbe
  1. Run bhe <stage(s)> .
# prepare enviroment in control-proxy and cluster for all actions 
$ hbe init 

# install nessary libs in control-proxy for compiling...
$ hbe initcontrolp 

# initally compile source code, configure site, distribute binary libs into cluster, 
# prepare runtime environment for cluster
$ hbe initdeploy 

# prepare runtime environment for cluster 
$ hbe initcluster 

# initially compile source code in control-proxy.
# This stage will resolve maven depandency and download necessary jars.
$ hbe initcompile  
	
# compile source code, configure site, distribute binary libs into cluster
$ hbe deploy 

# configure site.xml, worker, hadoop-env.sh and sync into cluster ...
$ hbe config 

# compile source code in control-proxy. default compile hadoop-main.
# params: yapi, yclient, ycommon, yscommon, ysrm, ysnm
$ hbe compile 
              
# add permissions for stage-sync, and also create hdfs dirs ...
$ hbe syncp 

# distribute binary libs into cluster. default sync hadoop-main.
# params: yapi, yclient, ycommon, yscommon, ysrm, ysnm
$ hbe sync
           
# clean cluster files. 
# params: log
$ hbe clean 

# default start-all.sh. 
# params: yarn, hdfs
$ hbe start

# default stop-all.sh.
# params: yarn, hdfs 
$ hbe stop 

# submit applications into cluster. 
# The basepath of execution is cluster binary libs path.
$ hbe submit <ins1> <ins2>

# ========================EXAMPLES AS FOLLOWING======================== #

$ hbe initcompile # first compile

$ hbe initdeploy # first compile and deploy

$ hbe deploy

$ hbe compile && hbe stop && hbe sync && hbe config && hbe strart

$ hbe compile ysrm ysnm 

$ hbe sync ysrm ysnm 

$ hbe clean log

$ hbe submit "./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-beta1.jar pi -Dmapreduce.job.num-opportunistic-maps-percent='100' 50 50" "./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-beta1.jar pi -Dmapreduce.job.num-opportunistic-maps-percent='50' 100 100"

PSEUDO_DIS_MODE

run, compile, benchmark and all actions are only in your dev-PC.

           run|compile|bench|report
    user ------> |__|
            control-proxy-pc

FULLY_DIS_MODE

compile, view report are only in yout dev-PC.
run, benchmark, performance log are in cluster-PCs.

                                        run jobs/benchmark
            compile|report    deploy      |_|_|_|_|_|    
    user ------> |__| ----------------->  |_|_|_|_|_|
           control-proxy-pc               |_|_|_|_|_|
                                *.jar       cluster     

How to customize step(s) and organize stage(s)

Rules:

  1. put *.py into ./scripts/ and *.sh into ./utilities/
  2. customized python files need to inherit basis.py and overwrite its action() method.
  3. define tigger function to support automatical execution.

Notes

Git & Bash

Compile

References

Hadoop: BUILDING.txt

Hadoop: How To Contribute

Book: Apache Hadoop YARN

hadoop-build-env's People

Contributors

xshaun avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.