Please use this identifier to cite or link to this item:
Scopus Web of Science® Altmetric
Type: Journal article
Title: Standard setting in Australian medical schools
Author: Ward, H.
Chiavaroli, N.
Fraser, J.
Mansfield, K.
Starmer, D.
Surmon, L.
Veysey, M.
O'Mara, D.
Citation: BMC Medical Education, 2018; 18(1):80-1-80-9
Publisher: BioMed Central
Issue Date: 2018
ISSN: 1472-6920
Statement of
Helena Ward, Neville Chiavaroli, James Fraser, Kylie Mansfield, Darren Starmer, Laura Surmon, Martin Veysey and Deborah O'Mara
Abstract: Background: Standard setting of assessment is critical in quality assurance of medical programs. The aims of this study were to identify and compare the impact of methods used to establish the passing standard by the 13 medical schools who participated in the 2014 Australian Medical Schools Assessment Collaboration (AMSAC). Methods: A survey was conducted to identify the standard setting procedures used by participating schools. Schools standard setting data was collated for the 49 multiple choice items used for benchmarking by AMSAC in 2014. Analyses were conducted for nine schools by their method of standard setting and key characteristics of 28 panel members from four schools. Results: Substantial differences were identified between AMSAC schools that participated in the study, in both the standard setting methods and how particular techniques were implemented. The correlation between the item standard settings data by school ranged from − 0.116 to 0.632. A trend was identified for panel members to underestimate the difficulty level of hard items and overestimate the difficulty level of easy items for all methods. The median derived cut-score standard across schools was 55% for the 49 benchmarking questions. Although, no significant differences were found according to panel member standard setting experience or clinicians versus scientists, panel members with a high curriculum engagement generally had significantly lower expectations of borderline candidates (p = 0.044). Conclusion: This study used a robust assessment framework to demonstrate that several standard setting techniques are used by Australian medical schools, which in some cases use different techniques for different stages of their program. The implementation of the most common method, the Modified Angoff standard setting approach was found to vary markedly. The method of standard setting used had an impact on the distribution of expected minimally competent student performance by item and overall, with the passing standard varying by up to 10%. This difference can be attributed to the method of standard setting because the ASMSAC items have been shown over time to have consistent performance levels reflecting similar cohort ability. There is a need for more consistency in the method of standard setting used by medical schools in Australia.
Keywords: Standard setting; Assessment; Preclinical teaching; Medical education
Description: Published online: 23 April 2018
Rights: © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.
RMID: 0030086665
DOI: 10.1186/s12909-018-1190-6
Appears in Collections:Medical Education Unit publications

Files in This Item:
File Description SizeFormat 
hdl_124062.pdfPublished version873.05 kBAdobe PDFView/Open

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.