"Identifying struggling students in real-time provides a virtual learning environment with an opportunity to intervene meaningfully with supports aimed at improving student learning and engagement. In this paper, we present a detailed analysis of quit prediction modeling in students playing a learning game called Physics Playground. From the interaction log data of the game, we engineered a comprehensive set of aggregated features of varying levels of granularity and trained individualized level-specific models and a single level-agnostic model. Contrary to our initial expectation, our results suggest that a level-agnostic model achieves superior predictive performance. We enhanced this model further with level-related and student-related features, leading to a moderate increase in AUC. Visualizing this model, we observe that it is based on high-level intuitive features that are generalizable across levels. This model can now be used in future work to automatically trigger cognitive and affective supports to motivate students to pursue a game level until completion."
Karumbaiah, S., Baker, R.S., Shute, V. (2018) Predicting Quitting in Students Playing a Learning Game. Proceedings of the 11th International Conference on Educational Data Mining, 21-31 (nominated for best paper)
1. SUMMARY
Through studying student engagements with the game “Physics Playground”, the paper found that there are high-level intuitive features that can be used to build models to support and/or motivate students. Smart supports are important since it can be harmful if supports like scaffolding are provided when students are doing well. The scope includes 57 male, 80 female in 8th and 9th grades, spans over four consecutive days with one pretest, two days of game play, and one post test. Classifiers were built for each level and for all levels (agnostic model) using gradient boosting algorithms using scikit-learn (python). Five-fold cross-validation was used for evaluating models. Area under the curve (AUC) was used as the evaluation metric.
The best approach is to start with the level-agnostic model with features common across all levels and then enhance it by feature additions when the model is run on a specific level. Specifically, 34 among 101 features were selected by the final model with 21 student-related features. Top 5 features include: number of visits to a particular level (the more visits, the less quit), standard deviation of total time spent, mean number of achievement badge received by all students and by a particular student (less badge, more difficult level, more quits), number of past quits.
2. STRENGTHS
The paper was very well-constructed and demonstrated a good depth of knowledge into the topic of using data analytic techniques to predict the outcome of a phenomenon. The main insight is about the right strategy to build a data-driven prediction system that can predict certain student behaviors. It is best to start from the most shared features among levels and then move on to enhancing the general model with selective level-specific features when targeting a specific level.
3. WEAKNESSES
It should be obvious that 101 features are too much if we compare this amount with that of more complicated prediction projects done by other teams in the past. The paper wanted to predict the behavior in real time but most of the data in top 15 features have update-cycles longer than what should be used for real-time predictions.
Image credit: The Chronicle of Higher Education
