The role of actively open-minded thinking in information acquisition, accuracy, and calibration

Judgment and Decision Making, Vol. 8, No. 3, May 2013, pp. 188–201 The role of actively open-minded thinking in information acquisition, accuracy, and calibration Uriel Haran∗ Ilana Ritov† Barbara A. Mellers‡ Abstract Errors in estimating and forecasting often result from the failure to collect and consider enough relevant information. We examine whether attributes associated with persistence in information acquisition can predict performance in an estimation task. We focus on actively open-minded thinking (AOT), need for cognition, grit, and the tendency to maximize or satisfice when making decisions. In three studies, participants made estimates and predictions of uncertain quantities, with varying levels of control over the amount of information they could collect before estimating. Only AOT predicted performance. This relationship was mediated by information acquisition: AOT predicted the tendency to collect information, and information acquisition predicted performance. To the extent that available information is predictive of future outcomes, actively open-minded thinkers are more likely than others to make accurate forecasts. Keywords: forecasting, prediction, overconfidence, calibration, individual differences, actively open-minded thinking. 1 Introduction ity dimensions that might be related to performance, and seek an explanation for how they work. Research in disciplines such as meteorology, statistics, finance, and psychology has tried to measure and explain the relationship between people’s confidence in their predictions and the accuracy of those predictions (e.g., Gigerenzer, Hoffrage, & Kleinbölting, 1991; Harvey, 1997; Henrion & Fischhoff, 1986; Klayman, Soll, González-Vallejo, & Barlas, 1999). Overconfidence in the accuracy of one’s estimates—sometimes called overprecision, to distinguish it from other types of overconfidence (Moore & Healy, 2008)—refers to the discrepancy between the confidence people have in the accuracy of their estimates, predictions, or beliefs and actual accuracy rate. Overconfidence has proven to be robust and difficult to remedy, although some interventions have been partially successful (Haran, Moore, & Morewedge, 2010; Soll & Klayman, 2004; Speirs-Bridge et al., 2010). In this work, we examine cognitive styles and personalWe thank Alon Mednikov, Talya Horowitz and Damaris Graeupner for help in data collection. This research was supported by the Intelligence Advanced Research Projects Activity (IARPA) via the Department of Interior National Business Center contract number D11PC20061. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions expressed herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government. Copyright: © 2013. The authors license this article under the terms of the Creative Commons Attribution 3.0 License. ∗ Ben-Gurion University of the Negev. Marcus Family Campus, Beer-Sheva, 8410501, Israel. Email: . † Hebrew University of Jerusalem. Email: . ‡ University of Pennsylvania. Email: . 1.1 Prediction error and insufficient search for information Most studies attribute confidence-accuracy miscalibration to one of two shortcomings. The first is the underappreciation of uncertainty and sources of error (e.g., Erev, Wallsten, & Budescu, 1994; Gigerenzer et al., 1991; Soll, 1996). Specifically, Juslin, Winman, and Hansson (2007) argued that judges make two errors in transforming samples of information into an estimate: they perceive the sample as an exact, unbiased representation of the estimated population; and they fail to acknowledge that sample variances are smaller than population variances. As a consequence, their estimates often miss the mark. The second shortcoming is the tendency to focus on the first answer that comes to mind, while failing to properly consider alternative outcomes (e.g., McKenzie, 1998). This failure to consider alternatives may come in the form of an incomplete search for relevant information, failure to retrieve available information from memory, or underweighting the importance or validity of information inconsistent with one’s initial hypothesis. The estimation process begins with a search in memory for relevant information to provide a tentative answer. This tentative answer, once reached, biases the search and retrieval of new information, as well as the interpretation of ambiguous evidence, in favor of the initial conclusion (e.g., Hoch, 1985; Koriat, Lichtenstein, & Fischhoff, 1980). 188 Judgment and Decision Making, Vol. 8, No. 3, May 2013 Building on this conceptualization, researchers have tried to improve confidence-accuracy calibration by encouraging judges to direct more attention to alternative evidence and other possible answers. Fischhoff and Bar Hillel (1984) instructed participants to look at the problems they were solving from different perspectives. Others (Hirt & Markman, 1995; Morgan & Keith, 2008) asked forecasters to project multiple scenarios, rather than imagine the one they deemed most probable. McKenzie (1997) explicitly told participants to take the alternative into account before making an estimate, whereas Koriat et al. (1980) instructed judges to generate self-contradicting arguments. These studies have reported modest success in reducing the discrepancy between the confidence judges displayed in their estimates and their accuracy, not by increasing accuracy, but by reducing confidence. Actively open-minded thinking 189 Is estimate quality an individual attribute? & Grosch, 1990). For example, some evidence indicates that men are more overconfident in their estimates than are women (Barber & Odean, 2001). Calibration is also related to expertise (Koehler, Brenner, & Griffin, 2002), though not in every estimate format (McKenzie, Liersch, & Yaniv, 2008). Surprisingly, not many relationships have been found between accurate estimations and personality attributes. Extraversion correlates negatively with accuracy and calibration on various cognitive and estimation tasks (Lynn, 1961; Schaefer, Williams, Goodie, & Campbell, 2004; Taylor & McFatter, 2003), but positively with short-term recall (Howarth & Eysenck, 1968; Osborne, 1972). McElroy and Dowd (2007) found that openness to experience was related to greater susceptibility to the anchoring bias. Finally, overconfidence has been linked to proactiveness (Pallier et al., 2002), narcissism (Campbell, Goodie, & Foster, 2004), selfmonitoring (Cutler & Wolfe, 1989), and trait optimism (Buehler & Griffin, 2003). Researchers have established a stronger link between cognitive style and estimation performance. For example, McElroy and Seta (2003) found that an analytic and systematic processing style correlated with reduced susceptibility t (...truncated)