Feedback on Feedback: comparing Classic Natural Language Processing and Generative AI to Evaluate Peer Feedback
Files
(Published version)
Date
2024
Authors
Hutt, S.
Depiro, A.
Wang, J.
Rhodes, S.
Baker, R.S.
Hieb, G.
Sethuraman, S.
Ocumpaugh, J.
Mills, C.
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Conference paper
Citation
Fourteenth International Conference On Learning Analytics and Knowledge, Lak 2024, 2024, pp.55-65
Statement of Responsibility
Conference Name
14th Annual International Conference on Learning Analytics and Knowledge (LAK) - Learning Analytics in the Age of Artificial Intelligence (18 Mar 2024 - 22 Mar 2024 : Kyoto, Japan)
Abstract
Peer feedback can be a powerful tool as it presents learning opportunities for both the learner receiving feedback as well as the learner providing feedback. Despite its utility, it can be difficult to implement effectively, particularly for younger learners, who are often novices at providing feedback. It can be difficult for students to learn what constitutes '' good '' feedback - particularly in openended problem-solving contexts. To address this gap, we investigate both classical natural language processing techniques and large language models, specifically ChatGPT, as potential approaches to devise an automated detector of feedback quality (including both student progress towards goals and next steps needed). Our findings indicate that the classical detectors are highly accurate and, through feature analysis, we elucidate the pivotal elements influencing its decision process. We find that ChatGPT is less accurate than classical NLP but illustrate the potential of ChatGPT in evaluating feedback, by generating explanations for ratings, along with scores. We discuss how the detector can be used for automated feedback evaluation and to better scaffold peer feedback for younger learners.
School/Discipline
Dissertation Note
Provenance
Description
Access Status
Rights
Copyright 2024 ACM (https://creativecommons.org/licenses/by-nc-sa/4.0/)
Access Condition Notes: This is an open access article