Variation observed in consensus judgements between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research
Date
2023
Authors
Kanukula, R.
McKenzie, J.E.
Cashin, A.G.
Korevaar, E.
McDonald, S.
Mello, A.T.
Nguyen, P.-Y.
Saldanha, I.J.
Wewege, M.A.
Page, M.J.
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Journal article
Citation
Journal of Clinical Epidemiology, 2023; 166:111244-1-111244-10
Statement of Responsibility
Raju Kanukula, Joanne E. McKenzie, Aidan G. Cashin, Elizabeth Korevaar, Sally McDonald, Arthur T. Mello, Phi-Yen Nguyen, Ian J. Saldanha, Michael A. Wewege, Matthew J. Page
Conference Name
Abstract
Objectives: To evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the Risk Of Bias due to Missing Evidence (ROB-ME) tool and determine inter-rater agreement in assessments. Study Design and Setting: We assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet’s Agreement Coefficient (AC) in consensus judgments between pairs. Results: Across the eight signaling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet’s AC ranged from 0.39 to 0.76. For the risk-of-bias judgment, the raw percentage agreement was 76% (95% confidence interval 60% to 92%) and Gwet’s AC was 0.47 (95% confidence interval 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as ‘‘low risk’’. Conclusion: Our findings indicated substantial variation in assessments in consensus judgments between pairs for the signaling questions and overall risk-of-bias judgments. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.
School/Discipline
Dissertation Note
Provenance
Description
Access Status
Rights
© 2024 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/).