Error running batch job with spark-submit about spark-kafka-cassandra-applying-lambda-architecture HOT 7 CLOSED

Can you please make sure you're running the latest version of the image and git scripts. They have both changed significantly in the last few days and weeks that it makes sense to just start there.

See which version of the vagrant image you have:
vagrant box list
Do you have 0.0.6 or something else?

If you have something older than 0.0.5 then let's get you on the latest box. Here's what to do (it will require some network bandwidth to download the new image) and this will permanently destroy the current one you have so be sure to backup or take a copy of anything you have that's on there.

Get latest from git, from within the directory spark-kafka-cassandra-applying-lambda-architecture

git pull origin master

Destroy old box, run from spark-kafka-cassandra-applying-lambda-architecture/vagrant directory

vagrant destroy

Get latest box version, run from spark-kafka-cassandra-applying-lambda-architecture/vagrant directory

vagrant box update

Start new vagrant box

vagrant up

Thanks

from spark-kafka-cassandra-applying-lambda-architecture.

Thanks! I get it working!

I checked that I had the latest box 0.0.6, but after destroying the old box and starting a new one, everything worked fine. It's kind of weird...oz..

Thanks again.

from spark-kafka-cassandra-applying-lambda-architecture.

Hi,

I failed when run batchjob to yarn
I already use boxes v0.0.6
and the errol log is:

`16/12/06 06:02:51 INFO yarn.Client: Application report for application_1480993486657_0008 (state: FAILED)
16/12/06 06:02:51 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1480993486657_0008 failed 2 times due to AM Container for appattempt_1480993486657_0008_000002 exited with exitCode: 10
For more detailed output, check application tracking page:http://lambda-pluralsight:8088/cluster/app/application_1480993486657_0008Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1480993486657_0008_02_000001
Exit code: 10
Stack trace: ExitCodeException exitCode=10:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 10
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1481004161092
final status: FAILED
tracking URL: http://lambda-pluralsight:8088/cluster/app/application_1480993486657_0008
user: vagrant
Exception in thread "main" org.apache.spark.SparkException: Application application_1480993486657_0008 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1034)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1081)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/12/06 06:02:51 INFO util.ShutdownHookManager: Shutdown hook called
16/12/06 06:02:51 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-08705c9f-0994-4da8-8b48-82eb9282313f
`

from spark-kafka-cassandra-applying-lambda-architecture.

There isn't enough information in the logs provided here to nail down the cause of the problem.
Would you mind checking the fixes.sh file under the vagrant directory. In the fixes.sh file, look for a section called: # spark-defaults

If you don't see it, then you simply need to update the project from git and do a vagrant reload --provision like so:

git pull origin master
vagrant reload --provision

That should take care of it. The problem this fixes is that the spark defaults were too high for the very limited resources the VM is working with so that sections takes care of adding some defaults that should work for everyone and it results in editing the file /pluralsight/spark/conf/spark-defaults.conf

from spark-kafka-cassandra-applying-lambda-architecture.

Closing this issue as original poster has this resolved now. @azzam-krya , if you're still having problems, ensure you're running the code as the root user. To get root, run the following.
sudo su -

If you have further problems, please open another ticket and kindly provide the following:

• User used to run command
• Command used
• Provide full error log. To do so. Follow these steps:

1. Browse to http://lambda-pluralsight:8088/
2. Click on the "History" link of the application that failed in the rightmost column under Tracking UI
3. Click on Logs link under Logs column
4. Click on the stderr link (first link)
5. That will show a page with a message up top "Showing ... bytes. Click here for full log" .. Go ahead and click on the "here" link
6. Copy all the logs from there and provide as input for the ticket.

from spark-kafka-cassandra-applying-lambda-architecture.

it works after I restart vm. thanks

from spark-kafka-cassandra-applying-lambda-architecture.

I am having a similar problem as this one and raised this ticket for it today -as I didnt see this ticket previously.

#27

from spark-kafka-cassandra-applying-lambda-architecture.