I'm quite confused what's your goal? Run hivebench concurrently or just run workload join or aggregation only once at a time? If you prefer the latter, no metastore service need to be run.

Make sure you do the maven download first and clean derby folders if needed, then do the preparing and running.

from hibench.

Hello thank you for the reply.I want to run both join and aggregation
workload ( each at a time). Your help is appreciated to the below questions:

1. Should running mvn process-sources produce a failure in the Reactor
   Summary for hivebench(please see below):
   [INFO]
   [INFO] hibench ........................................... SUCCESS [0.002s]
   [INFO] nutchindexing ..................................... SUCCESS [13.185s]
   [INFO] mahout ............................................ SUCCESS [4.328s]
   [INFO] hivebench ......................................... FAILURE [1.153s]

2.What is the command to clean derby folders. A simple /HiBench/bin
derby.log produces(see below):
ubuntu@master:~/HiBench/bin$ derby.log
-bash: /home/ubuntu/hadoop/bin/derby.log: Permission denied

3.Running prepare-concurrent.sh produces "command not found". Am i missing
something such as "start" before the command or am doing it wrong.
ubuntu@master:~/HiBench/bin$ prepare-concurrent.sh
prepare-concurrent.sh: command not found

On Sun, Dec 21, 2014 at 9:01 PM, Neo Qian [email protected] wrote:

I'm quite confused what's your goal? Run hivebench concurrently or just
run workload join or aggregation only once at a time? If you prefer the
latter, no metastore service need to be run.

Make sure you do the maven download first and clean derby folders if
needed, then do the preparing and running.

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

1. The hivebench should get success result in running mvn process-sources
2. Just remove folders named metastore_db, sometimes it matters
3. No need to run prepare-concurrent.sh in your case, you can locate in hivebench/bin folder and run prepare.sh, then run-aggregation.sh or run-join.sh

from hibench.

I don't know why hivebench is getting failure. I re-installed but problem
still exit. I believe that's why the commands don't work for hivebench
because of failure during mvn process-sources. Any help on how to rectify
this?

On Mon, Dec 22, 2014 at 10:25 PM, Neo Qian [email protected] wrote:

The hivebench should get success result in running mvn process-sources
2.

Just remove folders named metastore_db, sometimes it matters
3.

No need to run prepare-concurrent.sh in your case, you can locate in
hivebench/bin folder and run prepare.sh, then run-aggregation.sh or
run-join.sh

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

Can you display the error info during mvn process-sources?

from hibench.

Sir below is the error info during mvm oprocess-sources

[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] hibench ........................................... SUCCESS [0.002s]
[INFO] nutchindexing ..................................... SUCCESS [11.993s]
[INFO] mahout ............................................ SUCCESS [1.429s]
[INFO] hivebench ......................................... FAILURE [0.872s]

[INFO]

[INFO] BUILD FAILURE

[INFO]

[INFO] Total time: 14.650s
[INFO] Finished at: Wed Dec 24 11:09:29 UTC 2014
[INFO] Final Memory: 8M/20M

[INFO]

[ERROR] Failed to execute goal
com.googlecode.maven-download-plugin:download-maven-plugin:1.2.0:wget
(default) on project hivebench: IO Error: Could not get content -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal com.googlecode.maven-download-plugin:download-maven-plugin:1.2.0:wget
(default) on project hivebench: IO Error
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: IO Error
at com.googlecode.WGet.execute(WGet.java:260)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: org.apache.maven.plugin.MojoFailureException: Could not get
content
at com.googlecode.WGet.execute(WGet.java:248)
... 21 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn -rf :hivebench
ubuntu@master:~/HiBench/common/hibench$

On Tue, Dec 23, 2014 at 7:38 PM, Neo Qian [email protected] wrote:

Can you show the error info during mvn process-sources?

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

Maybe you can change a repo by modifying common/hibench/hivebench/pom.xml.

from hibench.

Sir,
I get a success now after modifying common/hibench/hivebench/pom.xml to
my hive repo. I will go ahead and run the commands based on our previous
email exchange and update you accordingly. Thanks for running through my
errors with me. Happy New Year
[INFO] Reactor Summary:
[INFO]
[INFO] hibench ........................................... SUCCESS [0.002s]
[INFO] nutchindexing ..................................... SUCCESS [13.646s]
[INFO] mahout ............................................ SUCCESS [4.290s]
[INFO] hivebench ......................................... SUCCESS [1.293s]

On Tue, Dec 30, 2014 at 9:20 PM, Neo Qian [email protected] wrote:

Maybe you can change a repo by modifying common/hibench/hivebench/pom.xml.

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

I located to hivebench/bin folder and I did run prepare.sh but i got
"command not found" (below)
ubuntu@master:/HiBench/hivebench/bin$ prepare.sh
prepare.sh: command not found
ubuntu@master:/HiBench/hivebench/bin$

On Wed, Dec 31, 2014 at 12:42 PM, koffi Dossou [email protected] wrote:

Sir,
I get a success now after modifying common/hibench/hivebench/pom.xml to
my hive repo. I will go ahead and run the commands based on our previous
email exchange and update you accordingly. Thanks for running through my
errors with me. Happy New Year
[INFO] Reactor Summary:
[INFO]
[INFO] hibench ........................................... SUCCESS [0.002s]
[INFO] nutchindexing ..................................... SUCCESS
[13.646s]
[INFO] mahout ............................................ SUCCESS [4.290s]
[INFO] hivebench ......................................... SUCCESS [1.293s]

On Tue, Dec 30, 2014 at 9:20 PM, Neo Qian [email protected]
wrote:

Maybe you can change a repo by modifying common/hibench/hivebench/pom.xml
.

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

from hibench.

I located to hivebench/bin folder and I did run prepare.sh but i got
"command not found" (below)
ubuntu@master:/HiBench/hivebench/bin$ prepare.sh
prepare.sh: command not found
ubuntu@master:/HiBench/hivebench/bin$

On Wed, Dec 31, 2014 at 12:52 PM, koffi Dossou [email protected] wrote:

I located to hivebench/bin folder and I did run prepare.sh but i got
"command not found" (below)
ubuntu@master:/HiBench/hivebench/bin$ prepare.sh
prepare.sh: command not found
ubuntu@master:/HiBench/hivebench/bin$

On Wed, Dec 31, 2014 at 12:42 PM, koffi Dossou [email protected]
wrote:

Sir,
I get a success now after modifying common/hibench/hivebench/pom.xml to
my hive repo. I will go ahead and run the commands based on our previous
email exchange and update you accordingly. Thanks for running through my
errors with me. Happy New Year
[INFO] Reactor Summary:
[INFO]
[INFO] hibench ........................................... SUCCESS
[0.002s]
[INFO] nutchindexing ..................................... SUCCESS
[13.646s]
[INFO] mahout ............................................ SUCCESS
[4.290s]
[INFO] hivebench ......................................... SUCCESS
[1.293s]

On Tue, Dec 30, 2014 at 9:20 PM, Neo Qian [email protected]
wrote:

Maybe you can change a repo by modifying
common/hibench/hivebench/pom.xml.

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

from hibench.

You should use
./prepare.sh
instead of
prepare.sh

from hibench.

ok I got it after i run ./prepare.sh. looks like am getting a new error
ubuntu@master:~/HiBench/hivebench/bin$ ./prepare.sh
========== preparing hive data ==========
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 39:
/home/hadoop/b
in/hadoop: No such file or directory
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 41:
/home/hadoop/b
in/hadoop: No such file or directory
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 43:
/home/hadoop/b
in/hadoop: No such file or directory
JAVA_HOME=/usr/lib/jvm/java-7-oracle
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 60:
HADOOP_HOME:
/home/ubuntu/hadoop

On Sun, Jan 4, 2015 at 8:29 PM, Daoyuan Wang [email protected]
wrote:

You should use
./prepare.sh
instead of
prepare.sh

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

You should set your HADOOP_HOME in hibench-config.sh before run hibench.
For the details, please refer to https://github.com/intel-hadoop/HiBench#getting-started

from hibench.

After I set my HADOOP_HOME in hibench-config.sh am still getting error
running ./prepare.sh
ubuntu@master:/HiBench/hivebench/bin$ ./prepare.sh
========== preparing hive data ==========
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 39: version:
command not found
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 41: version:
command not found
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 43: version:
command not found
JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_PREFIX=/home/ubuntu/hadoop
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 61:
HADOOP_EXECUTABLE: HADOOP_PREFIX/bin
ubuntu@master:/HiBench/hivebench/bin$

**below is how i set my paths in
home/ubuntu/HiBench/bin/hibench-config.sh

echo JAVA_HOME=${JAVA_HOME:?/usr/lib/jvm/java-7-oracle}
echo HADOOP_PREFIX=${HADOOP_PREFIX:?/home/ubuntu/hadoop}
echo HADOOP_EXECUTABLE=${HADOOP_EXECUTABLE:?HADOOP_PREFIX/bin}
echo HADOOP_CONF_DIR=${HADOOP_CONF_DIR:?HADOOP_PREFIX/conf}
echo HADOOP_EXAMPLES_JAR=${HADOOP_EXAMPLES_JAR:?HADOOP_PREFIX}
echo MAPRED_EXECUTABLE=${MAPRED_EXECUTABLE:?/home/ubuntu/hadoop}

On Mon, Jan 5, 2015 at 3:38 AM, Daoyuan Wang [email protected]
wrote:

You should set your HADOOP_HOME in hibench-config.sh before run hibench.
For the details, please refer to
https://github.com/intel-hadoop/HiBench#getting-started

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

you are pointing to the wrong HADOOP_EXECUTABLE, this should be at HADOOP_HOME/bin/hadoop

from hibench.

Sir, Am still getting new error even after pointing HADOOP_EXECUTABLE
to HADOOP_HOME/bin/hadoop.
Your help is appreciated

ubuntu@master:/HiBench/hivebench/bin$ ./prepare.sh
========== preparing hive data ==========
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 39:
HADOOP_HOME: command not found
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 41:
HADOOP_HOME: command not found
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 43:
HADOOP_HOME: command not found
JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_HOME=home/ubuntu/hadoop
HADOOP_EXECUTABLE=HADOOP_HOME/bin/hadoop
HADOOP_CONF=HADOOP_HOME/conf/hadoop
HADOOP_EXAMPLES=HADOOP_HOME/conf/hadoop
MAPRED_EXECUTABLE=HADOOP_HOME/conf/hadoop
./prepare.sh: line 46: HADOOP_HOME: command not found
ubuntu@master:/HiBench/hivebench/bin$

On Tue, Jan 6, 2015 at 8:46 PM, Daoyuan Wang [email protected]
wrote:

you are pointing to the wrong HADOOP_EXECUTABLE, this should be at
HADOOP_HOME/bin/hadoop

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

You have to set HADOOP_HOME first...

from hibench.

I did set HADOOP_HOME by pointing it to my hadoop location which is
home/ubuntu/hadoop. Is that what you mean by setting HADOOP_HOME first?
###################### Global Paths ##################

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export HADOOP_HOME=home/ubuntu/hadoop
export HADOOP_EXECUTABLE=HADOOP_HOME/bin/hadoop
export HADOOP_CONF=HADOOP_HOME/conf/hadoop
export HADOOP_EXAMPLES=HADOOP_HOME/conf/hadoop
export MAPRED_EXECUTABLE=HADOOP_HOME/conf/hadoop

On Wed, Jan 7, 2015 at 7:30 PM, Daoyuan Wang [email protected]
wrote:

You have to set HADOOP_HOME first...

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

you are setting all of these wrong...
suppose your HADOOP_HOME is at /home/ubuntu/hadoop

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export HADOOP_HOME=home/ubuntu/hadoop
export HADOOP_EXECUTABLE=$HADOOP_HOME/bin/hadoop # add $ first
export HADOOP_CONF=$HADOOP_HOME/conf # this should point to the directory containing those xml configures
export HADOOP_EXAMPLES= # this point to your hadoop_examples jar
export MAPRED_EXECUTABLE=HADOOP_HOME/bin/mapred # if in hadoop-1, use the same as HADOOP_EXECUTABLE

from hibench.

Sir honestly speaking i followed your instruction in your previous e-mail
and set path but am still getting error. Looks like
home/ubuntu/hadoop/bin/hadoop does not exit but in
home/ubuntu/hadoop/bin i see a file called hadoop in there so am lost as
to why its saying no such file or directory?Your continuous help is
appreciated.
ubuntu@master:/HiBench/hivebench/bin$ ./prepare.sh
========== preparing hive data ==========
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 40:
home/ubuntu/hadoop/bin/hadoop: No such file or directory
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 42:
home/ubuntu/hadoop/bin/hadoop: No such file or directory
/home/ubuntu/HiBench/hivebench/../bin/hibench-config.sh: line 44:
home/ubuntu/hadoop/bin/hadoop: No such file or directory
JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_HOME=home/ubuntu/hadoop
HADOOP_EXECUTABLE=home/ubuntu/hadoop/bin/hadoop
HADOOP_CONF=home/ubuntu/hadoop/conf
HADOOP_EXAMPLES=home/ubuntu/hadoop/hadoop-examples*.jar
MAPRED_EXECUTABLE=home/ubuntu/hadoop/bin/hadoop # using hadoop-1 so i set
it same as HADOOP_EXECUTABLE
./prepare.sh: line 46: home/ubuntu/hadoop/bin/hadoop: No such file or
directory
ubuntu@master:/HiBench/hivebench/bin$

On Wed, Jan 7, 2015 at 8:41 PM, Daoyuan Wang [email protected]
wrote:

you are setting all of these wrong...
suppose your HADOOP_HOME is at /home/ubuntu/hadoop

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export HADOOP_HOME=home/ubuntu/hadoop
export HADOOP_EXECUTABLE=$HADOOP_HOME/bin/hadoop # add $ first
export HADOOP_CONF=$HADOOP_HOME/conf # this should point to the directory
containing those xml configures
export HADOOP_EXAMPLES= # this point to your hadoop_examples jar
export MAPRED_EXECUTABLE=HADOOP_HOME/bin/mapred # if in hadoop-1, use the
same as HADOOP_EXECUTABLE

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

oh, I missed a slash there, sorry.
export HADOOP_HOME=/home/ubuntu/hadoop

from hibench.

works now but after running ./prepare.sh and then ./run-aggregation.sh.i
get some error from hive bin but then i received a success from
dependencies when i runed mvn process-sources earlier
========== running hive-aggregate bench ==========
Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_HOME=/home/ubuntu/hadoop
HADOOP_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
HADOOP_CONF=/home/ubuntu/hadoop/conf
HADOOP_EXAMPLES=/home/ubuntu/hadoop/hadoop-examples*.jar
MAPRED_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
Error: The hive bin file hasn't be downloaded by maven, please check!
ubuntu@master:~/HiBench/hivebench/bin$ ./run-aggregation.sh
========== running hive-aggregate bench ==========
Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_HOME=/home/ubuntu/hadoop
HADOOP_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
HADOOP_CONF=/home/ubuntu/hadoop/conf
HADOOP_EXAMPLES=/home/ubuntu/hadoop/hadoop-examples*.jar
MAPRED_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
Error: The hive bin file hasn't be downloaded by maven, please check!

On Thu, Jan 8, 2015 at 11:44 PM, Daoyuan Wang [email protected]
wrote:

oh, I missed a slash there, sorry.
export HADOOP_HOME=/home/ubuntu/hadoop

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

@qiansl127 Can you take a look?

from hibench.

Please check if HiBench/common/hibench/hivebench/target/hive-0.12.0-bin.tar.gz exists. After the success of mvn process-sources, the tar file should already be downloaded.

from hibench.

I have a working hive on my cluster that I use regularly but locating to .
HiBench/common/hibench/hivebench/target. I find a single hive file which
contains(below):

Index of /apache/hive

 Name
                 Last modified      Size  Description
 Parent Directory
                            -
 hive-0.13.1/            30-Oct-2014 19:44    -
 hive-0.14.0/            12-Nov-2014 16:21    -
 stable/

On Fri, Jan 9, 2015 at 7:48 AM, Neo Qian [email protected] wrote:

> Please check if
> HiBench/common/hibench/hivebench/target/hive-0.12.0-bin.tar.gz exists.
> After the success of mvn process-sources, the tar file should already be
> downloaded.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/intel-hadoop/HiBench/issues/73#issuecomment-69335964.

from hibench.

If you want to use your own hive bin, you can modify $HIVE_HOME and $HIVE_RELEASE in HiBench/bin/hibench-config.sh.

from hibench.

Should my HIVE_RELEASE be set to my Hive path which is /home/ubuntu/hive?
Am using hive 0.9.0 in my cluster.Upon downloading hive a couple of months
back i moved hive-0.9.0-bin to hive in unix. Should i then modify HIVE_HOME
and HIVE_RELEASE like so (below)
if [ -z "$HIVE_HOME" ]; then
export HIVE_HOME=hive-0.9.0-bin
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${home/ubuntu/hive}

On Sun, Jan 11, 2015 at 7:45 PM, Neo Qian [email protected] wrote:

If you want to use your own hive bin, you can modify $HIVE_HOME and
$HIVE_RELEASE in HiBench/bin/hibench-config.sh

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

Am still getting the same error I think am not setting the path to Hive
right?

JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_HOME=/home/ubuntu/hadoop
HADOOP_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
HADOOP_CONF=/home/ubuntu/hadoop/conf
HADOOP_EXAMPLES=/home/ubuntu/hadoop/hadoop-examples*.jar
MAPRED_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
Error: The hive bin file hasn't be downloaded by maven, please check!

**hivebench.config.sh
[ -z "$HIVE_HOME" ]; then
export HIVE_HOME=/home/ubuntu/hive
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}
fi

On Tue, Jan 13, 2015 at 6:09 AM, koffi Dossou [email protected] wrote:

Should my HIVE_RELEASE be set to my Hive path which is /home/ubuntu/hive?
Am using hive 0.9.0 in my cluster.Upon downloading hive a couple of months
back i moved hive-0.9.0-bin to hive in unix. Should i then modify HIVE_HOME
and HIVE_RELEASE like so (below)
if [ -z "$HIVE_HOME" ]; then
export HIVE_HOME=hive-0.9.0-bin
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${home/ubuntu/hive}

On Sun, Jan 11, 2015 at 7:45 PM, Neo Qian [email protected]
wrote:

If you want to use your own hive bin, you can modify $HIVE_HOME and
$HIVE_RELEASE in HiBench/bin/hibench-config.sh

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

from hibench.

@qiansl127 https://github.com/qiansl127
@adrian-wang

Hello Qian and Wang,
this is my hive configuration in my
bashrc(below) it will be nice to know how i can infuse this configuration
into $HIVE_HOME and $HIVE_RELEASE in HiBench/bin/hibench-config.sh. the
hive in bashrc the one i use for hive computation. Your prompt reply is
appreciated to enable me complete this research that am doing at the
graduate level.
#hive environment Varaibles
export HIVE_CONF=/home/ubuntu/hive/conf
export HIVE_PREFIX=/home/ubuntu/hive
export PATH=$PATH:$HIVE_PREFIX/bin

On Wed, Jan 14, 2015 at 5:45 AM, koffi Dossou [email protected] wrote:

Am still getting the same error I think am not setting the path to Hive
right?

JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_HOME=/home/ubuntu/hadoop
HADOOP_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
HADOOP_CONF=/home/ubuntu/hadoop/conf
HADOOP_EXAMPLES=/home/ubuntu/hadoop/hadoop-examples*.jar
MAPRED_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
Error: The hive bin file hasn't be downloaded by maven, please check!

**hivebench.config.sh
[ -z "$HIVE_HOME" ]; then
export HIVE_HOME=/home/ubuntu/hive
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}
fi

On Tue, Jan 13, 2015 at 6:09 AM, koffi Dossou [email protected] wrote:

Should my HIVE_RELEASE be set to my Hive path which is /home/ubuntu/hive?
Am using hive 0.9.0 in my cluster.Upon downloading hive a couple of months
back i moved hive-0.9.0-bin to hive in unix. Should i then modify HIVE_HOME
and HIVE_RELEASE like so (below)
if [ -z "$HIVE_HOME" ]; then
export HIVE_HOME=hive-0.9.0-bin
export
HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${home/ubuntu/hive}

On Sun, Jan 11, 2015 at 7:45 PM, Neo Qian [email protected]
wrote:

If you want to use your own hive bin, you can modify $HIVE_HOME and
$HIVE_RELEASE in HiBench/bin/hibench-config.sh

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

from hibench.

Can you use HIVE_HOME instead of HIVE_PREFIX?

from hibench.

oh yeah i changed it back to HIVE_HOME and still error exit. i think the
problem is changing $HIVE_HOME and $HIVE_RELEASE in
HiBench/bin/hibench-config.sh
as Neo said . In my previous email I made some assumption in HiBench/bin/
hibench-config.sh as how i feel it should look but still was getting the
same error?

On Thu, Jan 15, 2015 at 8:18 PM, Daoyuan Wang [email protected]
wrote:

Can you use HIVE_HOME instead of HIVE_PREFIX?

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

You should set HIVE_HOME in hibench-config.sh, instead of your bashrc.

from hibench.

This is how I set it in hivebench.config.sh (below). The bashrc i showed
is an already installed hive i have which I changed the hive_prefix to
hive_home
[ -z "$HIVE_HOME" ]; then
export HIVE_HOME=/home/ubuntu/hive
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}

On Thu, Jan 15, 2015 at 11:38 PM, Daoyuan Wang [email protected]
wrote:

You should set HIVE_HOME in hibench-config.sh, instead of your bashrc.

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

I think you need to remove
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}
This aims to find the hive that maven downloaded previously. If you want to use your own hive, you shouldn't include that line.

from hibench.

Ok I did that . After I removed export
HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}.
I was getting the same error so i went ahead to re-run mvn process-sources so
i can get the dependencies but i got an error .error(below) thanks.
ubuntu@master:~/HiBench/common/hibench$ mvn process-sources -e
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[ERROR] The build could not read 1 project -> [Help 1]
org.apache.maven.project.ProjectBuildingException: Some problems were
encountered while processing the POMs:
[ERROR] Child module /home/ubuntu/HiBench/common/hibench/hivebench/pom.xml
of /home/ubuntu/HiBench/common/hibench/pom.xml does not exist @

    at

org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:363)
at
org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636)
at
org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
[ERROR]
[ERROR] The project com.intel.hibench:hibench:4.0-SNAPSHOT
(/home/ubuntu/HiBen
[ERROR] Child module
/home/ubuntu/HiBench/common/hibench/hivebench/pom.xml o
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please rea
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildin
ubuntu@master:~/HiBench/common/hibench$ mvn process-sources
[INFO] Scanning for projects...
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project com.intel.hibench:hibench:4.0-SNAPSHOT
(/home/ubuntu/HiBen
ch/common/hibench/pom.xml) has 1 error
[ERROR] Child module
/home/ubuntu/HiBench/common/hibench/hivebench/pom.xml o
f
/home/ubuntu/HiBench/common/hibench/pom.xml does not exist
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please rea
d the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildin

gException
u

On Fri, Jan 16, 2015 at 11:27 AM, Daoyuan Wang [email protected]
wrote:

I think you need to remove
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}
This aims to find the hive that maven downloaded previously. If you want
to use your own hive, you shouldn't include that line.

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

This is what I changed in *
/home/ubuntu/HiBench/common/**hibench/pom.xml *based
on error in previous email.I tried to set repo to my hive location.
.

http://mirrors.ibiblio.org
apache/home/ubuntu/hive/bin

On Fri, Jan 16, 2015 at 1:44 PM, koffi Dossou [email protected] wrote:

Ok I did that . After I removed export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}.
I was getting the same error so i went ahead to re-run mvn
process-sources so i can get the dependencies but i got an error
.error(below) thanks.
ubuntu@master:~/HiBench/common/hibench$ mvn process-sources -e
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[ERROR] The build could not read 1 project -> [Help 1]
org.apache.maven.project.ProjectBuildingException: Some problems were
encountered while processing the POMs:
[ERROR] Child module /home/ubuntu/HiBench/common/hibench/hivebench/pom.xml
of /home/ubuntu/HiBench/common/hibench/pom.xml does not exist @

    at

org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:363)
at
org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636)
at
org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
[ERROR]
[ERROR] The project com.intel.hibench:hibench:4.0-SNAPSHOT
(/home/ubuntu/HiBen
[ERROR] Child module
/home/ubuntu/HiBench/common/hibench/hivebench/pom.xml o
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please rea
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildin
ubuntu@master:~/HiBench/common/hibench$ mvn process-sources
[INFO] Scanning for projects...
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project com.intel.hibench:hibench:4.0-SNAPSHOT
(/home/ubuntu/HiBen
ch/common/hibench/pom.xml) has 1 error
[ERROR] Child module
/home/ubuntu/HiBench/common/hibench/hivebench/pom.xml o
f
/home/ubuntu/HiBench/common/hibench/pom.xml does not exist
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the
-e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please rea
d the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildin

gException
u

On Fri, Jan 16, 2015 at 11:27 AM, Daoyuan Wang [email protected]
wrote:

I think you need to remove
export HIVE_HOME=${DEPENDENCY_DIR}/hivebench/target/${HIVE_HOME}
This aims to find the hive that maven downloaded previously. If you want
to use your own hive, you shouldn't include that line.

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

from hibench.

If you decide to use your own hive, then leave the mvn command, that is meant to download a hive and some other dependencies for other workloads.
For hivebench, just make sure everything is configured right and then start your workload.

from hibench.

I run ./run-aggregation.sh and had an output but how do I make sense of
what has been generated (see below please) to incorporate into my working
paper.I understand from the output that it took 1 mapper and 1 reducer to
complete a job using CPU time of 2.0sec and time of 0.333 is that correct?

ubuntu@master:~/HiBench/hivebench/bin$ ./run-aggregation.sh
========== running hive-aggregate bench ==========
Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

JAVA_HOME=/usr/lib/jvm/java-7-oracle
HADOOP_HOME=/home/ubuntu/hadoop
HADOOP_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
HADOOP_CONF=/home/ubuntu/hadoop/conf
HADOOP_EXAMPLES=/home/ubuntu/hadoop/hadoop-examples*.jar
MAPRED_EXECUTABLE=/home/ubuntu/hadoop/bin/hadoop
Warning: $HADOOP_HOME is deprecated.

rm: cannot remove -r: No such file or directory.
rm: cannot remove -skipTrash: No such file or directory.
rm: cannot remove /user/hive/warehouse/uservisits_aggre: No such file or
directo
ry.
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
use or
g.apache.hadoop.log.metrics.EventCounter in all the
log4j.properties files.
Logging initialized using configuration in
jar:file:/home/ubuntu/hive/lib/hive-c

ommon-0.9.0.jar!/hive-log4j.properties
Hive history file=/tmp/ubuntu/hive_job_log_ubuntu_201501202005_582129071.txt
OK
Time taken: 6.149 seconds
OK
Time taken: 1.273 seconds
OK
Time taken: 0.018 seconds
OK
Time taken: 0.325 seconds
OK
Time taken: 0.033 seconds
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_201412202223_0080, Tracking URL =
http://master:50030/jobdetails.jsp?jobid=job_201412202223_0080
Kill Command = /home/ubuntu/hadoop/libexec/../bin/hadoop job
-Dmapred.job.tracker=hdfs://master:8021 -kill job_201412202223_0080
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 1
2015-01-20 20:05:31,335 Stage-1 map = 0%, reduce = 0%
2015-01-20 20:05:34,351 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:35,356 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:36,360 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:37,365 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:38,369 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:39,373 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:40,377 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:41,382 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:42,387 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.86 sec
2015-01-20 20:05:43,402 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
2.0 sec
2015-01-20 20:05:44,405 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
2.0 sec
2015-01-20 20:05:45,410 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
2.0 sec
2015-01-20 20:05:46,421 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
2.0 sec
MapReduce Total cumulative CPU time: 2 seconds 0 msec
Ended Job = job_201412202223_0080
Loading data to table default.uservisits_aggre
Deleted hdfs://master:8020/user/hive/warehouse/uservisits_aggre
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Cumulative CPU: 2.0 sec HDFS Read: 524 HDFS
Write: 130 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 0 msec
OK
Time taken: 20.62 seconds
(standard_in) 1: syntax error
Warning: $HADOOP_HOME is deprecated.

(standard_in) 1: syntax error
Warning: $HADOOP_HOME is deprecated.

rm: cannot remove -r: No such file or directory.
rm: cannot remove -skipTrash: No such file or directory.
rm: cannot remove /HiBench/Hive/Output-comp/uservisits_aggre: No such file
or directory.
Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

On Tue, Jan 20, 2015 at 12:36 AM, Daoyuan Wang [email protected]
wrote:

If you decide to use your own hive, then leave the mvn command, that is
meant to download a hive and some other dependencies for other workloads.
For hivebench, just make sure everything is configured right and then
start your workload.

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

did you run prepare before you run aggregation?

from hibench.

Is there any update?

from hibench.

Everything is working great. Upon configuring additional nodes of 4 - 16
(for the time being), I see the time change from 0.0034sec for a 4 node to
0.049sec for a 16 node run-join workload and for a run-aggregation 4node
(0.03sec) - 16node (0.05).Also the total CPU time changed immensely across
various number of nodes with 16 node having the highest for a run-join
workload of 12sec720msec though other nodes shows a decrease in CPU time
that is 4node(17sec 520msec), 8 node(7sec 250msec),12 node (1sec
670msec).I'll be happy to email you the outputs of each workload for your
input as well. I was expecting a time decrease as I added more nodes but
that has been conflicting since(4 node) has lower time than a 16 node. Your
take on this please

On Fri, Feb 13, 2015 at 12:04 AM, Daoyuan Wang [email protected]
wrote:

Is there any update?

—
Reply to this email directly or view it on GitHub
#73 (comment).

from hibench.

If the data set is not so large but you force it to run on a large cluster, there would be too much networking and IO consuming. Since the origin issue is gone, I am going to close this one.
Thanks for your feedback, and your interest in HiBench!

from hibench.

[INFO] Reactor Summary:
[INFO]
[INFO] hibench ........................................... SUCCESS [0.939s]
[INFO] nutchindexing ..................................... SUCCESS [12.502s]
[INFO] mahout ............................................ SUCCESS [15.796s]
[INFO] hivebench ......................................... SUCCESS [4.656s]
[INFO] Sparkbench ........................................ SUCCESS [50.538s]
[INFO] PEGASUS: A Peta-Scale Graph Mining System ......... SUCCESS [4.649s]
[INFO] HiBench data generation tools ..................... SUCCESS [14.058s]
[INFO] Streamingbench .................................... SUCCESS [0.012s]
[INFO] Streamingbench data generation tools .............. SUCCESS [11.365s]
[INFO] Streaming Benchmark Zookeeper Helper Utils ........ SUCCESS [9.765s]
[INFO] Streaming Benchmark Spark ......................... SUCCESS [44.975s]
[INFO] Streaming Benchmark Storm ......................... FAILURE [1.745s]
[INFO] Streaming Benchmark Samza ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:53.167s
[INFO] Finished at: Sun Nov 22 11:15:21 IST 2015
[INFO] Final Memory: 73M/288M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.thrift.tools:maven-thrift-plugin:0.1.11:compile (thrift-sources) on project streaming-bench-storm: thrift did not exit cleanly. Review output for more information. -> [Help 1]

How to solve this?? Please give your valuable suggession........

from hibench.

@attu-mittu , please open a new issue for any issue you encounter next time.
Please refer to #147 to resolve your problem.

from hibench.

okay thank you. :)

from hibench.