Online Prediction Accuracy about edm2016 HOT 2 OPEN

To your first question, having no data about a situation is always tricky. How you solve this depends greatly on your application and how you perceive the costs of being wrong. Concepts you might find interesting to study related to this problem are the exploration-exploitation tradeoff and zero shot learning. Also, there is specific literature around IRT and flashcard applications, as well as large data sets available from flashcard applications to try out any ideas you might have retrospectively.

To your second question, the parameters for splitting schemes used in the paper appear in the README of the repo. We used a five-fold scheme, and reported the average of our metrics.

from edm2016.

Thanks for the clarification; just a few follow up questions:

1. What are your thoughts on training a model with pre-existing data? For instance, DKT/logistic models do not require any student/item specific parameters, hence, the model could be trained with data collected elsewhere, and then implemented in an application to make immediate, accurate predictions.

For a model that does not require item or student parameters, would this be appropriate? What are the benefits of using data from the same students/items to train general weights? A model trained on dataset A could then be tested on datasets B and C, just like a real flashcard scenario; what are your thoughts on this?

2. When trying to evaluate an IRT model through online prediction accuracy, after determining all item parameters, is the ability parameter updated through retraining the model with ALL the data collected thus far (all the students + training data), or just the students’ INDIVIDUAL data? In other words, what data is used to train the student-level parameters?

Thanks again, looking forward to your response.

from edm2016.