Standard setting of script concordance tests using an adapted Nedelsky approach

Abstract

BACKGROUND: Standard setting in assessment seeks to apply meaning of achievement to an assessment score. Appropriate standard setting for script concordance tests (SCTs) remains a challenge, with existing methods representing norm-referenced approaches. AIMS: To develop a criterion-referenced standard setting approach for SCT using an adapted Nedelsky approach, to pilot feasibility, and to compare failure rates with two other methods. METHODS: Second- and third-year medical students were administered a 45-question SCT and results collated. Standard setting was applied using three approaches: (1) norm-referenced (student cohorts), (2) expert-referenced (student cohort compared to expert mean), and (3) adapted Nedelsky approach using answer key normalization. Feasibility and failure rates were measured. RESULTS: All standard setting approaches were feasible, with 60 additional minutes required for the Nedelsky standard setting exercise. Failure rates between the three approaches were similar (Year 2: 8.0–9.8% and Year 3: 2.1–7.6%), with the adapted Nedelsky approach representing an intermediate option (Year 2: 8.0% and Year 3: 3.5%). CONCLUSION: Standard setting SCT using the criterion-referenced method of an adapted Nedelsky approach was found to be both logically justifiable and logistically simple, and produced failure rates comparable to other currently utilized and less objective approaches.