"A key affordance of game-based learning environments is their potential to unobtrusively assess student learning without interfering with gameplay. In this paper, we introduce a temporal analytics framework for stealth assessment that analyzes students’ problem-solving strategies. The strategy-based temporal analytic framework uses long short-term memory network-based evidence models and clusters sequences of students’ problem-solving behaviors across consecutive tasks. We investigate this strategybased temporal analytics framework on a dataset of problemsolving behaviors collected from student interactions with a gamebased learning environment for middle school computational thinking. The results of an evaluation indicate that the strategybased temporal analytics framework significantly outperforms competitive baseline models with respect to stealth assessment predictive accuracy."
Akram, Bita, Wookhee Min, Eric N. Wiebe, Bradford W. Mott, Kristy Boyer and James C. Lester. “Improving Stealth Assessment in Game-based Learning with LSTM-based Analytics.” EDM (2018).
1. SUMMARY
Because students behaviors unfolds over time and because it is very challenging to predict students’ performance based on raw data, the paper built and assess a temporal analytics framework for stealth assessment that is strategy-based using long short-term memory networks. This framework utilized both students’ problem-solving behavior traces and pre-test rather than just pre-test data as in other methods. Temporal dependencies in dynamic behaviors were captured to form interaction patterns. Evaluation results shown that this approach significantly outperforms other competitive baseline models within the aspect of predictive accuracy.
2. STRENGTHS
The paper has a good scope of 244 students spanning four public middle school classrooms, the educational software was well-built (an important factor in the quality of the experiment). Clustering appears to be a good strategy for identifying groups of students’ strategies. The paper also differentiate itself from others by eliminating some features such as the number of retry attempts. Data were presented neatly using meaningful charts. A combination of various machine learning algorithms were used (random forrest, SVM, LSTM).
3. WEAKNESSES
Clustering method may have issues with precision and the paper admits that by suggesting deep-learning method as future research direction. In such case, cost in money and development time can be an issue.
The paper proposed an interesting approach and is very relevant to current trends.
