"Automated writing evaluation (AWE) is a popular form of educational technology designed to supplement writing instruction and feedback, yet research on the effectiveness of AWE has observed mixed findings. The current study considered how students' perceptions of automated essay scoring and feedback influenced their writing performance, revising behaviors, and future intentions toward the technology. The manner in which the software was presented—claims about the accuracy and quality of the automated scoring and feedback—were modestly related to students' expectations and perceptions. However, students' direct experiences with the software were most strongly associated with their perceptions. Importantly, students' perceptions seemed to have minimal impact on their “in the moment” use of the software to write and revise successfully. Students revised and improved their essays regardless of their positive or negative views of the system. However, positive and negative perceptions significantly predicted future intentions to use the software again or to recommend the software to a friend. Implications for AWE design, implementation, and evaluation are discussed."
Rod D.Roscoea,JoshuaWilsonb,Adam C.Johnsona, Christopher R. Mayra
1. SUMMARY
The paper studies Automated Writing Evaluation (AWE) and how students perceptions about the software will influence performances. The scope involves 110 under-grad students in a Psychology class. 69% of them are fluent in English. With the assistance of W-pal, students were asked to perform argument based writing. Additionally, they were randomly assigned to four groups informed with different manipulated information on system capabilities. For example, one group may be told the automated scoring was “under development” while it was told as “well-established” to another group. Essay quality was scored by W-pal algorithms. Results show that: (i) students are cautiously positive about W-pal scoring and feedback; (ii) Presented scoring and feedback quality have little to no impact on initial expectations, (iii) Direct experience appear to have most influences on students’ perceptions of AWE, (iv) Perceptions about scoring accuracy and feedback quality are not the same.
2. STRENGTHS
This is a solid paper in terms of both content and structure. The paper touches an interesting topic which belongs to both education and human-computer domains. Supported evidences were well presented. Results were well discussed and limitations were honestly laid out.
3. WEAKNESSES
The paper admits that collected metrics are not multidimensional. There is also no way to track if students following or not-following of recommendations are intentional or not. It is also better to evaluate students in more rounds with more time spent in order to fit real-life writing activities.
