A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models
Date
2025
Authors
Hoang, G.B.
Ransom, K.J.
Stephens, R.
Semmler, C.
Fay, N.
Mitchell, L.
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Conference paper
Citation
Workshop Proceedings of the 19th International AAAI Conference on Web and Social Media (ICWSM 2025), 2025, pp.1-12
Statement of Responsibility
Gia Bao Hoang, Keith J. Ransom, Rachel G. Stephens, Carolyn Semmler, Nicolas Fay, Lewis Mitchell
Conference Name
19th International AAAI Conference on Web and Social Media (ICWSM) (23 Jun 2025 - 26 Jun 2025 : Copenhagen, Denmark)
DOI
Abstract
Traditional psychological models of belief revision focus on face-to-face interactions, but with the rise of social media, more effective models are needed to capture belief revision at scale, in this rich text-based online discourse. Here, we use a hybrid approach, utilizing large language models (LLMs) to develop a model that predicts successful persuasion using features derived from psychological experiments. Our approach leverages LLM generated ratings of features previously examined in the literature to build a random forest classification model that predicts whether a message will result in belief change. Of the eight features tested, epistemic emotion and willingness to share to share were the top-ranking predictors of belief change in the model. Our findings provide insights into the characteristics of persuasive messages and demonstrate how LLMs can enhance models of successful persuasion based on psychological theory. Given these insights, this work has broader applications in fields such as online influence detection and misinformation mitigation, as well as measuring the effectiveness of online narratives.
School/Discipline
Dissertation Note
Provenance
Description
Workshop: NLPSI 2025: First Workshop on Integrating NLP and Psychology to Study Social Interactions. Workshops held on June 23rd, 2025.
Access Status
Rights
© 2025, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.