Kanukula, R.McKenzie, J.E.Cashin, A.G.Korevaar, E.McDonald, S.Mello, A.T.Nguyen, P.-Y.Saldanha, I.J.Wewege, M.A.Page, M.J.2024-02-012024-02-012023Journal of Clinical Epidemiology, 2023; 166:111244-1-111244-100895-43561878-5921https://hdl.handle.net/2440/140398Objectives: To evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the Risk Of Bias due to Missing Evidence (ROB-ME) tool and determine inter-rater agreement in assessments. Study Design and Setting: We assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet’s Agreement Coefficient (AC) in consensus judgments between pairs. Results: Across the eight signaling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet’s AC ranged from 0.39 to 0.76. For the risk-of-bias judgment, the raw percentage agreement was 76% (95% confidence interval 60% to 92%) and Gwet’s AC was 0.47 (95% confidence interval 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as ‘‘low risk’’. Conclusion: Our findings indicated substantial variation in assessments in consensus judgments between pairs for the signaling questions and overall risk-of-bias judgments. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.en© 2024 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/).Bias; Reporting bias; Meta-analysis; Nutritional sciences; Systematic review; ReliabilityJournal article10.1016/j.jclinepi.2023.1112442024-01-29679817Kanukula, R. [0000-0003-0793-786X]