dearleiii / dblp_with_mongodb Goto Github PK

dearleiiis-MacBook-Pro:~ dearleiii$ sbt package
[info] Loading project definition from /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/project
[info] Loading settings for project spark from build.sbt ...
[info] Set current project to SparkApp (in build file:/Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/)
[info] Compiling 1 Scala source to /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/target/scala-2.10/classes ...
[error] /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/src/main/scala/MongoSpark.scala:75:22: overloaded method constructor MongoUpdateWritable with alternatives:
[error]   (x$1: org.bson.BasicBSONObject,x$2: org.bson.BasicBSONObject,x$3: Boolean,x$4: Boolean,x$5: Boolean)com.mongodb.hadoop.io.MongoUpdateWritable <and>
[error]   (x$1: org.bson.BasicBSONObject,x$2: org.bson.BasicBSONObject)com.mongodb.hadoop.io.MongoUpdateWritable <and>
[error]   ()com.mongodb.hadoop.io.MongoUpdateWritable
[error]  cannot be applied to (com.mongodb.BasicDBObject, com.mongodb.BasicDBObject, Boolean, Boolean)
[error]             value => new MongoUpdateWritable(
[error]                      ^
[error] /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/src/main/scala/MongoSpark.scala:84:17: value saveAsNewAPIHadoopFile is not a member of org.apache.spark.rdd.RDD[(Object, Nothing)]
[error]         updates.saveAsNewAPIHadoopFile(
[error]                 ^
[error] two errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 4 s, completed Nov 28, 2018, 4:47:49 PM

[info] Loading settings for project spark from build.sbt ...
[info] Set current project to SparkApp (in build file:/Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/)
[info] Compiling 1 Scala source to /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/target/scala-2.10/classes ...
[error] /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/src/main/scala/MongoSpark.scala:164:35: value reduceByKey is not a member of org.apache.spark.rdd.RDD[String]
[error]         val countRDD = authorsRDD.reduceByKey((a, b) => a+b)
[error]                                   ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 4 s, completed Nov 29, 2018, 11:37:16 AM
dearleiiis-MacBook-Pro:~ dearleiii$ sbt package
[info] Loading project definition from /Users/dearleiii/Desktop

[error] /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/src/main/scala/MongoSpark.scala:162:10: value flatmap is not a member of org.apache.spark.rdd.RDD[Any]
[error] possible cause: maybe a semicolon is missing before `value flatmap'?
[error]         .flatmap(authors => oneAuthor)
[error]          ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed

dearleiiis-MacBook-Pro:~ dearleiii$ sbt package
[info] Loading project definition from /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/project
[info] Loading settings for project spark from build.sbt ...
[info] Set current project to SparkApp (in build file:/Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/)
[info] Compiling 1 Scala source to /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/target/scala-2.10/classes ...
[error] /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/src/main/scala/MongoSpark.scala:75:22: overloaded method constructor MongoUpdateWritable with alternatives:
[error]   (x$1: org.bson.BasicBSONObject,x$2: org.bson.BasicBSONObject,x$3: Boolean,x$4: Boolean,x$5: Boolean)com.mongodb.hadoop.io.MongoUpdateWritable <and>
[error]   (x$1: org.bson.BasicBSONObject,x$2: org.bson.BasicBSONObject)com.mongodb.hadoop.io.MongoUpdateWritable <and>
[error]   ()com.mongodb.hadoop.io.MongoUpdateWritable
[error]  cannot be applied to (com.mongodb.BasicDBObject, com.mongodb.BasicDBObject, Boolean, Boolean)
[error]             value => new MongoUpdateWritable(
[error]                      ^
[error] /Users/dearleiii/Desktop/2018Autumn/DataBase/Homeworks/hw3/hw3_code/spark/src/main/scala/MongoSpark.scala:84:17: value saveAsNewAPIHadoopFile is not a member of org.apache.spark.rdd.RDD[(Object, Nothing)]
[error]         updates.saveAsNewAPIHadoopFile(
[error]                 ^
[error] two errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 4 s, completed Nov 28, 2018, 4:47:49 PM

val authorRDD = authorsRDD.map(line => line.split(","))
       println("print out each author: \n")
       authorRDD.take(10).foreach(indvArray => indvArray.foreach(println))

[ "Kien A. Hua" , "Chiang Lee"]
[ "Tiziana Catarci" , "Giovanna D'Angiolini" , "Maurizio Lenzerini"]
[ "Ada Wai-Chee Fu" , "Eamonn J. Keogh" , "Leo Yung Hang Lau" , "Chotirat (Ann) Ratanamahatana"]
[ "Jay-Louise Weldon"]
[ "Yan Qi 0002" , "K. Selçuk Candan" , "Maria Luisa Sapino"]
[ "Michael Steinbrunn" , "Klaus Peithner" , "Guido Moerkotte" , "Alfons Kemper"]
[ "Shu Lin" , "M. Tamer Özsu" , "Vincent Oria" , "Raymond T. Ng"]
[ "Richard J. Lipton" , "Jeffrey F. Naughton"]
[ "Joseph Albert"]
[ "Arvind Arasu" , "Venkatesh Ganti" , "Raghav Kaushik"]
print out each author:

[ "Kien A. Hua"
 "Chiang Lee"]
[ "Tiziana Catarci"
 "Giovanna D'Angiolini"
 "Maurizio Lenzerini"]
[ "Ada Wai-Chee Fu"
 "Eamonn J. Keogh"
 "Leo Yung Hang Lau"
 "Chotirat (Ann) Ratanamahatana"]
[ "Jay-Louise Weldon"]
[ "Yan Qi 0002"
 "K. Selçuk Candan"
 "Maria Luisa Sapino"]
[ "Michael Steinbrunn"
 "Klaus Peithner"
 "Guido Moerkotte"
 "Alfons Kemper"]
[ "Shu Lin"
 "M. Tamer Özsu"
 "Vincent Oria"
 "Raymond T. Ng"]
[ "Richard J. Lipton"
 "Jeffrey F. Naughton"]
[ "Joseph Albert"]
[ "Arvind Arasu"
 "Venkatesh Ganti"
 "Raghav Kaushik"]
dearleiiis-MacBook-P