aaron7 / lima Goto Github PK
View Code? Open in Web Editor NEWGroup Project Lima
Group Project Lima
After 5-10 mins of running the job on the HDFS local system - around 100% CPU on all 4 cores 6GB memory used:
Gets to the final reduce and gives this error
13/02/25 00:23:18 INFO mapreduce.Job: map 90% reduce 28%
13/02/25 00:23:21 INFO mapreduce.Job: map 94% reduce 28%
13/02/25 00:23:22 INFO mapreduce.Job: map 100% reduce 28%
13/02/25 00:23:22 INFO mapreduce.Job: Task Id : attempt_1361746790989_0001_r_000000_0, Status : FAILED
Error: java.lang.ClassCastException: class uk.ac.cam.cl.groupproject12.lima.hadoop.DosJob$DoSAttack
at java.lang.Class.asSubclass(Class.java:3049)
at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(SequenceFileAsBinaryOutputFormat.java:86)
at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getRecordWriter(SequenceFileAsBinaryOutputFormat.java:123)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:494)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:419)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
13/02/25 00:23:23 INFO mapreduce.Job: map 100% reduce 0%
13/02/25 00:23:26 INFO mapreduce.Job: Task Id : attempt_1361746790989_0001_r_000000_1, Status : FAILED
Error: java.lang.ClassCastException: class uk.ac.cam.cl.groupproject12.lima.hadoop.DosJob$DoSAttack
at java.lang.Class.asSubclass(Class.java:3049)
at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(SequenceFileAsBinaryOutputFormat.java:86)
at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getRecordWriter(SequenceFileAsBinaryOutputFormat.java:123)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:494)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:419)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
13/02/25 00:23:32 INFO mapreduce.Job: Task Id : attempt_1361746790989_0001_r_000000_2, Status : FAILED
Error: java.lang.ClassCastException: class uk.ac.cam.cl.groupproject12.lima.hadoop.DosJob$DoSAttack
at java.lang.Class.asSubclass(Class.java:3049)
at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(SequenceFileAsBinaryOutputFormat.java:86)
at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getRecordWriter(SequenceFileAsBinaryOutputFormat.java:123)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:494)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:419)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
13/02/25 00:23:36 INFO mapreduce.Job: map 100% reduce 100%
13/02/25 00:23:36 INFO mapreduce.Job: Job job_1361746790989_0001 failed with state FAILED due to:
13/02/25 00:23:36 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=2400
FILE: Number of bytes written=1986816
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1295056907
HDFS: Number of bytes written=0
HDFS: Number of read operations=60
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Failed reduce tasks=4
Launched map tasks=20
Launched reduce tasks=4
Data-local map tasks=20
Total time spent by all maps in occupied slots (ms)=9574328
Total time spent by all reduces in occupied slots (ms)=1199720
Map-Reduce Framework
Map input records=8749823
Map output records=4083
Map output bytes=661630
Map output materialized bytes=669916
Input split bytes=2240
Combine input records=0
Spilled Records=4083
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=20825
CPU time spent (ms)=260980
Physical memory (bytes) snapshot=5161984000
Virtual memory (bytes) snapshot=21500129280
Total committed heap usage (bytes)=4141350912
File Input Format Counters
Bytes Read=1295054667
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:891)
at uk.ac.cam.cl.groupproject12.lima.hadoop.RunDosJob.main(RunDosJob.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.