Sensitivity to hypothesis size during information search

Abstract

It is well known that people attempting to perform hypothesis testing show a positive test bias, preferring to request evidence that is consistent (rather than inconsistent) with their current hypothesis. Rather than viewing this as an irrational bias, information theoretic accounts of hypothesis testing have argued that selecting tests likely to produce positive evidence is adaptive when most hypotheses are small (i.e., true of few entities in the world) and respond positively to very few queries. These accounts make the prediction that as hypotheses get larger, the relative utility of positive evidence will decrease; when hypotheses are large enough, negative evidence will become more useful than positive evidence. We test if people are sensitive to this change in utility with an experiment inspired by the game “Battleship,” in which people attempt to discover the correct arrangement of ships by asking for positive or negative evidence. As predicted, as hypotheses become larger people request less positive evidence, and when hypotheses are large requests for negative evidence are more likely than requests for positive evidence. Implications for the nature of the positive test bias are discussed.