Impact of adjusting for inter-rater variability in conference abstract ranking and selection processes
Files
Date
2018
Authors
Scanlan, J.N.
Lannin, N.A.
Hoffmann, T.
Stanley, M.
Mcdonald, R.
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Journal article
Citation
Australian Occupational Therapy Journal, 2018; 65(1):54-62
Statement of Responsibility
Conference Name
Abstract
Background/aim: Scientific conferences provide a forum for clinicians, educators, students and researchers to share research findings. To be selected to present at a scientific conference, authors must submit a short abstract which is then rated on its scientific quality and professional merit and is accepted or rejected based on these ratings. Previous research has indicated that inter-rater variability can have a substantial impact on abstract selection decisions. For their 2015 conference, the Occupational Therapy Australia National Conference introduced a system to identify and adjust for inter-rater variability in the abstract ranking and selection process.
Method: Ratings for 1340 abstracts submitted for the 2015 and 2017 conferences were analysed using many-faceted Rasch analysis to identify and adjust for inter-rater variability. Analyses of the construct validity of the abstract rating instrument and rater consistency were completed. To quantify the influence of inter-rater variability of abstract selection decisions, comparisons were made between decisions made using Rasch-calibrated measure scores and decisions that would have been made based purely on raw average scores derived from the abstract ratings.
Results: Construct validity and measurement properties of the abstract rating tool were good to excellent (item fit MnSq scores ranged from 0.8 to 1.2; item reliability index = 1.0). Most raters (24 of 27, 89%) were consistent in their use of the rating instrument. When comparing abstract allocations under the two conditions, 25% of abstracts (n = 341) would have been allocated differently if inter-rater variability was not accounted for.
Conclusion: This study demonstrates that, even with a strong abstract rating instrument and a small rater pool, inter-rater variability still exerts a substantial influence on abstract selection decisions. It is recommended that all occupational therapy conferences internationally, and scientific conferences more generally, adopt systems to identify and adjust for the impact of inter-rater variability in abstract selection processes.
School/Discipline
Dissertation Note
Provenance
Description
Access Status
Rights
Copyright 2017 Occupational Therapy Australia
Access Condition Notes: Accepted manuscript available after 1 January 2019