Performance of context aware algorithms about carskit HOT 5 CLOSED

Hello, Thanks for your interests in our toolkit.

Actually, it is not surprising for me to see these results. Let's me explain the results from the perspective of FACT and EXPERIMENT:

FACT:
1). Whether context is able to improve recommendation, it is a domain-specific and data-specific problem. Therefore, it is not necessary to say that context-aware algorithms always outperform non-contextual algorithms;
2). Context selection is important in context-aware recsys. Including irrelevant context informtion in the data will introduce noises which impact the performance of context-aware recsys.
3). Context-aware data set is usually small and sparse, which leads to unreliable results in prediction errors. Usually we trust results in top-N recommendation instead of prediction errors.
3). Similarity-based context-aware recsys, such as CAMF_ICS, LCS, MCS, we view them as algorithms for top-N recommendation task. In addition, the splitting approaches is a kind of context-aware algorithms.
4). Some algorithms (e.g., CPTF and FM) in our library did not well treat the outbound problems in the learning process which leads to bad results in prediction errors, but still good results in top-N recommendation.

EXPERIMENT:
1). As mentioned before, you should try to select the most relevant and influential context dimensions in the preprocessing stage
2). You should tune up the paramters, especially for the learning-based algorithms
3). We do not trust error-based metrics on small data sets. Take the average-based algorithms for example, it may outperform a complex algorithm when data is very sparsy. Use may just rate an item for limited times, e.g., twice, then the average rating of these becomes the best choice. Apparently, it is unreliable to trust an average rating based on just limited number of rating profiles. So, that's the general problem in the context-aware recommendation.
4). In terms of the top-N recommendation, the evaluation on context-aware recsys is different from traditional ones, since we are going to recommend a list of items to <user, item, context>. For a small data set, user may just rate limited number of items in a sepcific context situation, so the results on top-N recommendation is usually very small compared with traditional recsys

Anyway, in short, your results are not surprising. And usually we do not trust the error-based metrics in this domain. We prefer top-N recommendations. In addition, you'd better try a larger data set and perform context selection before using it.

from carskit.

Hi,

Thanks a lot for the detailed answer. I was wondering about the performance of UserItemAvg as well. I will redo my experiments with ranking metrics and report my observations.

I have gone through the page you maintain for contextual datasets (thanks for that!); based on your experience which dataset do you believe would clearly showcase the improved performance of context aware recsys? I understand there would be some amount of data cleaning involved but if possible I would like to do that with a dataset that is more likely to show contextual differentiation.

from carskit.

This is a list of results I ran for the DePaulMovie data. I did not run with finer-grained parameters tunes, but only adjust the learning-rate for matrix-factorization based algorithms. And the evaluation is based on prediction errors. And we focus on the RMSE metric.

In my evaluation, the best non-contextual recommender is BiasedMF which obtains 0.964 on RMSE; the best context-aware algorithm is CAMF_CU which obtains a 0.884 result on RMSE.

Final Results by GlobalAvg, MAE: 1.229522, RMSE: 1.414676, NAME: 0.307381, rMAE: 1.237230, rRMSE: 1.452487, MPE: 0.000000, carskit.alg.baseline.avg.GlobalAverage@5fdef03a, Time: '00:00','00:00'
Final Results by UserAvg, MAE: 1.109387, RMSE: 1.316039, NAME: 0.277347, rMAE: 1.092272, rRMSE: 1.349704, MPE: 0.000000, carskit.alg.baseline.avg.UserAverage@5fdef03a, Time: '00:00','00:00'
Final Results by ItemAvg, MAE: 1.089705, RMSE: 1.300056, NAME: 0.272426, rMAE: 1.072582, rRMSE: 1.334432, MPE: 0.000000, carskit.alg.baseline.avg.ItemAverage@5fdef03a, Time: '00:00','00:00'
Final Results by UserKNN, MAE: 0.950417, RMSE: 1.170172, NAME: 0.237604, rMAE: 0.924645, rRMSE: 1.201833, MPE: 0.000000, 10, COS, -1, Time: '00:00','00:00'
Final Results by ItemKNN, MAE: 0.967005, RMSE: 1.195632, NAME: 0.241751, rMAE: 0.951093, rRMSE: 1.233333, MPE: 0.000000, 10, COS, -1, Time: '00:00','00:00'
Final Results by BiasedMF, MAE: 0.666487, RMSE: 0.964239, NAME: 0.166622, rMAE: 0.643873, rRMSE: 0.998125, MPE: 0.000000, numFactors: 20, numIter: 100, lrate: 0.002, maxlrate: -1.0, regB: 0.001, regU: 0.001, regI: 0.001, regC: 0.001, isBoldDriver: true, Time: '00:00','00:00'
Final Results by ItemSplitting-BiasedMF, MAE: 0.759239, RMSE: 0.978460, NAME: 0.189810, rMAE: 0.726789, rRMSE: 1.020789, MPE: 0.000000, numFactors: 10, numIter: 100, lrate: 2.0E-4, maxlrate: -1.0, regB: 0.001, regU: 0.001, regI: 0.001, regC: 0.001, isBoldDriver: true, Time: '00:01','00:00'
Final Results by UISplitting-BiasedMF, MAE: 0.767762, RMSE: 0.991922, NAME: 0.191940, rMAE: 0.734541, rRMSE: 1.030315, MPE: 0.000000, numFactors: 10, numIter: 100, lrate: 2.0E-4, maxlrate: -1.0, regB: 0.001, regU: 0.001, regI: 0.001, regC: 0.001, isBoldDriver: true, Time: '00:00','00:00'
Final Results by CAMF_C, MAE: 0.683617, RMSE: 0.929397, NAME: 0.170904, rMAE: 0.644069, rRMSE: 0.970799, MPE: 0.000000, numFactors: 10, numIter: 100, lrate: 0.002, maxlrate: -1.0, regB: 0.001, regU: 0.001, regI: 0.001, regC: 0.001, isBoldDriver: true, Time: '00:01','00:00'
Final Results by CAMF_CU, MAE: 0.637444, RMSE: 0.884012, NAME: 0.159361, rMAE: 0.596739, rRMSE: 0.918705, MPE: 0.000000, numFactors: 10, numIter: 100, lrate: 0.002, maxlrate: -1.0, regB: 0.001, regU: 0.001, regI: 0.001, regC: 0.001, isBoldDriver: true, Time: '00:01','00:00'
Final Results by CAMF_CI, MAE: 0.694390, RMSE: 0.930310, NAME: 0.173598, rMAE: 0.663152, rRMSE: 0.974802, MPE: 0.000000, numFactors: 10, numIter: 100, lrate: 2.0E-4, maxlrate: -1.0, regB: 0.001, regU: 0.001, regI: 0.001, regC: 0.001, isBoldDriver: true, Time: '00:02','00:00'

from carskit.

Hello, in my experience, you can find significant contextual effects on the most context-aware data sets listed in my repository, if you evaluate them by precison,recall,NDCG, MRR, MAP, etc.

from carskit.

Thanks for the numbers.

from carskit.