Invited commentary: Harnessing housing natural experiments is important, but beware differential misclassification of difference in difference

Abstract

In this issue of the Journal, Reeves et al. (Am J Epidemiol. 2016;184(6):421–429) present the findings of a natural experiment analyzing the association between reduced housing affordability and mental ill health. Their difference-in-difference analysis of cross-sectional, quarterly population health surveys administered before and after implementation of a policy to reduce Housing Benefit payments in the United Kingdom in April 2011 represents an important way to assess the impact of a national housing policy shift on public health. It is a well-conducted study harnessing a natural experiment and adds to the weight of evidence supporting an association between housing costs and mental health. However, quantitative bias analysis based on the reported findings suggests that a small amount of differential (by unblinded Housing Benefit status) misclassification bias in the outcome may be enough to explain the observed association. Our analysis of possible misclassification bias in the outcome used in the study highlights the need for caution when a difference-in-difference estimate is small, the population is not blinded to its postintervention exposure status, and the outcome measure is subjective and prone to differential (by unblinded exposure or treatment status) misclassification