Description:
Developed a scala maven program to monitor a folder in HDFS in real time such that any new file in the folder will be processed. For each RDD in the stream the following task will be perform,
- Count the word frequency and save the output in HDFS.
- Filter out the short words i.e <5 characters and save the output in HDFS.
- Count the co-occurence of words in each RDD where the context is the same line and save the output in the HDFS.
Requirements to run the program: Maven Java7 Scala
Steps to run the program:
-
Create the "Task2_Scala-1.0-SNAPSHOT-jar-with-dependencies" jar file by using maven build using clean and verify goal.
-
Upload the jar in Hue account using website.
-
Create a folder in the files named “input”.
-
Download the jar file in to the Hadoop server by using “hadoop fs -copyToLocal /user/Task2_Scala-1.0-SNAPSHOT-jar-with-dependencies.jar ~/"
-
To run the program execute the command in the Hadoop server "spark-submit --class streaming.NetworkWordCount --master yarn --deploy-mode client Task2_Scala-1.0-SNAPSHOT-jar-with-dependencies.jar hdfs:/user/input/ hdfs:/user/output2/sub_taskA_output hdfs:/user/output2/sub_taskB_output hdfs:/user/output2/sub_taskC_output"
-
To generate the output, upload the input files and wait for 10 seconds after uploading each input file in the "input" folder created in the step 3.