Variation observed in consensus judgements between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research

Simple item page
dc.contributor.authorKanukula, R.
dc.contributor.authorMcKenzie, J.E.
dc.contributor.authorCashin, A.G.
dc.contributor.authorKorevaar, E.
dc.contributor.authorMcDonald, S.
dc.contributor.authorMello, A.T.
dc.contributor.authorNguyen, P.-Y.
dc.contributor.authorSaldanha, I.J.
dc.contributor.authorWewege, M.A.
dc.contributor.authorPage, M.J.
dc.date.issued2023
dc.description.abstractObjectives: To evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the Risk Of Bias due to Missing Evidence (ROB-ME) tool and determine inter-rater agreement in assessments. Study Design and Setting: We assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet’s Agreement Coefficient (AC) in consensus judgments between pairs. Results: Across the eight signaling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet’s AC ranged from 0.39 to 0.76. For the risk-of-bias judgment, the raw percentage agreement was 76% (95% confidence interval 60% to 92%) and Gwet’s AC was 0.47 (95% confidence interval 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as ‘‘low risk’’. Conclusion: Our findings indicated substantial variation in assessments in consensus judgments between pairs for the signaling questions and overall risk-of-bias judgments. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.
dc.description.statementofresponsibilityRaju Kanukula, Joanne E. McKenzie, Aidan G. Cashin, Elizabeth Korevaar, Sally McDonald, Arthur T. Mello, Phi-Yen Nguyen, Ian J. Saldanha, Michael A. Wewege, Matthew J. Page
dc.identifier.citationJournal of Clinical Epidemiology, 2023; 166:111244-1-111244-10
dc.identifier.doi10.1016/j.jclinepi.2023.111244
dc.identifier.issn0895-4356
dc.identifier.issn1878-5921
dc.identifier.orcidKanukula, R. [0000-0003-0793-786X]
dc.identifier.urihttps://hdl.handle.net/2440/140398
dc.language.isoen
dc.publisherElsevier BV
dc.relation.granthttp://purl.org/au-research/grants/nhmrc/1139997
dc.relation.granthttp://purl.org/au-research/grants/nhmrc/GNT2009612
dc.relation.granthttp://purl.org/au-research/grants/nhmrc/GNT2010088
dc.relation.granthttp://purl.org/au-research/grants/arc/DE200101618
dc.rights© 2024 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/).
dc.source.urihttp://dx.doi.org/10.1016/j.jclinepi.2023.111244
dc.subjectBias; Reporting bias; Meta-analysis; Nutritional sciences; Systematic review; Reliability
dc.titleVariation observed in consensus judgements between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research
dc.typeJournal article
pubs.publication-statusPublished

Files

Collections

Public Health publications