For NCCU 1052 Data Science and Big Data Analytics Final Project
Finish a machine learning challenged hosted by Kaggle in conjunction with Expedia : Expedia-personalized-sort

1. Liu Z.Y.
2. Tammykan
3. Cwsu

1. visitor_hist_starrating
2. prpo_starrating
3. prop_review_score
4. prop_brand_bool
5. prop_location_score1
6. prop_location_score2
7. promotion_flag
8. orig_destination_distance

1. booking_bool

After download the data, I seperated the booking_true and booking_false data. And get the same number of rows with awk

    awk -F "," '{if(substr($54,0,1)=="1") {print}} ' data.csv > book.csv
    awk -F "," '{if(substr($54,0,1)=="0") {print}} ' data.csv > no_book.csv
    tail -n ? no_book.csv > no_book_tmp.csv
    cat book.csv no_book_tmp > data.csv

Use spark to run the python script

    spark-submit --master local ./Main.py

I use RandomForest.trainClassifier and the null model is 1/2(guess)

1. Accuracy: 77.89%
2. Area under Precision/Recall (PR) curve: 87%
3. Area under ROC curve: 78.251%

1. Spark doc
2. Personalize Expedia Hotel Searches - ICDM 2013
3. benhamner