Recommendation System Using Spark and System G

There are three parts of code:

(1). DataRetrieve.ipynb

It serves to preprocess the dataset.

Put amazon-meta.txt(input dataset) in the same directory of DataRetrieve.ipynb

Run the program The output will appear in the same directory with the name rater.txt

(2). spark_recommendation.py

It serves to do recommendation on spark.

Copy the path of your inputfile rater.txt

Replace filename with the path copied and save Run this python script with the command ./pyspark spark_recommendation.py It will output MSE, as well as the ten top recommendation results.

(3). system g part code(code is put in file 'step 3 systemg part code.pdf')

System G is used for item to item recommendation part by Colfilter function.

Using function analytic_colfilter, the analytic performs BFS for each vertex x up to --depth (must be an even number, default is 4), computes the number of paths N(x, y) from x to every vertex in Y, and ranks these vertices based on their N(x, y) values in a descending order.

So in our example, we will recommend top 10 items for each item in part 2.