Can Psychology Students and Researchers Distinguish between an Al-generated and a Human-authored Journal Abstract?
Date
2023
Authors
Siddique, Muhammad Omar
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Thesis
Citation
Statement of Responsibility
Conference Name
Abstract
Limited research has explored the application of Large Language Models (LLMs), such as ChatGPT, in writing journal abstracts. To determine the practicability of LLMs in psychology research writing, I investigated the distinguishability of Al-generated versus human-authored abstracts through a mixed between-within participants experiment. Fifty-six participants with varying levels of researcher experience in psychology completed a survey in which they viewed five Al-generated and five human-authored psychology abstracts. Using Signal Detection Theory (SDT) to investigate whether participants were able to distinguish between Al-generated and human-authored abstracts, it was found that students and researchers performed poorly when distinguishing between the two types of abstracts, with a mean performance of 52% correct and a d-prime of 0.11. I then investigated if the level of researcher experience between Undergraduate Students, Ongoing Postgraduate Students, and Completed Postgraduate Researchers, influenced ability to distinguish between Al-generated and human-authored abstracts. A one-way ANOVA and post-hoc tests revealed that Undergraduates and Ongoing Postgraduates were almost equally as proficient and demonstrated significantly better distinguishing ability than Completed Postgraduates. Finally, linvestigated whether participants evaluated the textual quality of ChatGPT abstracts as equivalent to human-authored abstracts and discovered through a paired t-test that Al-generated abstracts were evaluated as significantly better. The results present important implications regarding the use of LLMs within psychological research, as well as the need for students and researchers to build their Al literacy. Research protocol is available on the Open Science Framework (OSF). Key words: Artificial Intelligence (Al), Large Language Model (LLM), Abstracts, ChatGPT, Signal Detection Theory (SDT)
School/Discipline
School of Psychology
Dissertation Note
Thesis (B.PsychSc(Hons)) -- University of Adelaide, School of Psychology, 2023
Provenance
This electronic version is made publicly available by the University of Adelaide in accordance with its open access policy for student theses.
Copyright in this thesis remains with the author. This thesis may incorporate third party material which has been used by the author pursuant to Fair Dealing exceptions. If you are the author of this thesis and do not wish it to be made publicly available, or you are the owner of any included third party copyright material you wish to be removed from this electronic version, please complete the take down form located at: http://www.adelaide.edu.au/legals
Description
This item is only available electronically.