big-data-manning / big-data-code Goto Github PK
View Code? Open in Web Editor NEWSource code for Big Data: Principles and best practices of scalable realtime data systems
Source code for Big Data: Principles and best practices of scalable realtime data systems
Am facing issues in setting up Batch Layer successfully.
I introduced a main function in BatchWorkflow.java of manning.batchlayer package as follows:
{
initTestData();
batchWorkflow();
}
When I run this under hadoop (with 1 namenode and 2 datanodes), it fails at:
batchWorkflow()->ingest()->appendNewDataToMasterDataPail()->shred()->Api.execute()
with following output + exception:
15/07/17 10:16:20 INFO flow.FlowStep: [] submitted hadoop job: job_1437122465719 _0010
15/07/17 10:17:19 WARN flow.FlowStep: [] task completion events identify failed tasks
15/07/17 10:17:19 WARN flow.FlowStep: [] task completion events count: 5
15/07/17 10:17:19 WARN flow.FlowStep: [] event = Task Id : attempt_1437122465719_0010_m_000000_0, Status : SUCCEEDED
15/07/17 10:17:19 WARN flow.FlowStep: [] event = Task Id : attempt_1437122465719_0010_r_000000_0, Status : FAILED
15/07/17 10:17:19 WARN flow.FlowStep: [] event = Task Id : attempt_1437122465719_0010_r_000000_1, Status : FAILED
15/07/17 10:17:19 WARN flow.FlowStep: [] event = Task Id : attempt_1437122465719_0010_r_000000_2, Status : FAILED
15/07/17 10:17:19 WARN flow.FlowStep: [] event = Task Id : attempt_1437122465719_0010_r_000000_3, Status : TIPFAILED
15/07/17 10:17:19 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
15/07/17 10:17:19 INFO flow.Flow: [] stopping all jobs
15/07/17 10:17:19 INFO flow.FlowStep: [] stopping: (1/1) /tmp/swa/shredded
15/07/17 10:17:19 INFO impl.YarnClientImpl: Killed application application_1437122465719_0010
15/07/17 10:17:19 INFO flow.Flow: [] stopped all jobs
15/07/17 10:17:19 INFO util.Hadoop18TapUtil: deleting temp path /tmp/swa/shredded/_temporary
Exception in thread "main" cascading.flow.FlowException: step failed: (1/1) /tmp/swa/shredded, with job id: job_1437122465719_0010, please see cluster logs for failure messages
at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:193)
at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:137)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:122)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:42)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Any input to fix the issue will be appreciated. Thanks.
public static class AnalyzeVisits extends CascalogBuffer {
private static final int VISIT_LENGTH_SECS = 60 * 15;
public void operate(FlowProcess process, BufferCall call) {
Iterator<TupleEntry> it = call.getArgumentsIterator();
int bounces = 0;
int visits = 0;
Integer lastTime = null; <- lastTime initialized
int numInCurrVisit = 0;
while(it.hasNext()) {
TupleEntry tuple = it.next();
int timeSecs = tuple.getInteger(0);
if(lastTime == null ||
(timeSecs - lastTime) > VISIT_LENGTH_SECS) { <-- ??? when lastTime updated
visits++;
if(numInCurrVisit == 1) {
bounces++;
}
numInCurrVisit = 0;
}
numInCurrVisit++;
}
if(numInCurrVisit==1) {
bounces++;
}
call.getOutputCollector().add(new Tuple(visits, bounces));
}
}
Java noob here. Trying to compile and use Pail examples from Big Data book on Ubuntu 14.04. Below is the full output. Any and all help greatly appreciated! I'll continue to try and figure out. @handdola @nagarajanchinnasamy - kindly lend a hand? Cheers.
/home/bmeynell/big-data-code (master)$ ls
genthrift.sh pom.xml README.md src target
/home/bmeynell/big-data-code (master)$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
/home/bmeynell/big-data-code (master)$ mvn -version
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.7.0_101, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-57-generic", arch: "amd64", family: "unix"
/home/bmeynell/big-data-code (master)$ java -version
java version "1.7.0_101"
OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
/home/bmeynell/big-data-code (master)$ mvn compile
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building big-data-book 1.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ big-data-book ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/bmeynell/big-data-code/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ big-data-book ---
[INFO] Compiling 27 source files to /home/bmeynell/big-data-code/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.439s
[INFO] Finished at: Sun Oct 02 00:19:49 UTC 2016
[INFO] Final Memory: 14M/202M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project big-data-book: Compilation failure: Compilation failure:
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[36,1] error: annotations are not supported in -source 1.3
[ERROR]
[ERROR] (use -source 5 or higher to enable annotations)
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[38,66] error: generics are not supported in -source 1.3
[ERROR]
[ERROR] (use -source 5 or higher to enable generics)
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[50,21] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[50,63] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[56,25] error: for-each loops are not supported in -source 1.3
[ERROR]
[ERROR] (use -source 5 or higher to enable for-each loops)
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[91,4] error: invalid method declaration; return type required
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[104,22] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[105,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[106,146] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[107,15] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[108,66] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[108,86] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[108,104] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[111,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[117,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[120,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[122,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[125,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[128,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[133,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[137,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[140,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[143,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[148,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[151,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[154,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[156,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[157,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[159,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[163,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[164,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[167,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[170,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[171,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[174,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[177,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[181,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[182,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[184,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[186,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[188,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[190,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[192,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[194,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[197,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[199,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[201,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[202,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[203,2] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[205,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[213,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[215,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[221,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[223,4] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/ExternalDataSystem.java:[282,0] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[36,1] error: annotations are not supported in -source 1.3
[ERROR]
[ERROR] (use -source 5 or higher to enable annotations)
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[38,56] error: generics are not supported in -source 1.3
[ERROR]
[ERROR] (use -source 5 or higher to enable generics)
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[56,21] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[56,63] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,4] error: invalid method declaration; return type required
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,20] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,21] error: ')' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,26] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,27] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,28] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,29] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,47] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,48] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[57,49] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,10] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,11] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,12] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,17] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,18] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,20] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,29] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,30] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[62,31] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,10] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,11] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,12] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,17] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,18] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,20] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,29] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,30] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[63,31] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[68,25] error: for-each loops are not supported in -source 1.3
[ERROR]
[ERROR] (use -source 5 or higher to enable for-each loops)
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[109,4] error: invalid method declaration; return type required
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[124,23] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[125,10] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[126,22] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[127,9] error: class or interface expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[128,146] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[129,14] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[129,38] error: <identifier> expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[129,40] error: illegal start of type
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[129,43] error: ')' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[129,47] error: ';' expected
[ERROR]
[ERROR] /home/bmeynell/big-data-code/src/java/manning/schema/Pedigree.java:[129,85] error: <identifier> expected
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.