The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks

Maggie E. Toplak 0 1 Richard F. West 0 1 Keith E. Stanovich 0 1 0 K. E. Stanovich Department of Human Development and Applied Psychology, University of Toronto , Toronto , Ontario, Canada 1 R. F. West Department of Psychology, James Madison University , Harrisonburg, VA, USA 2 ) Department of Psychology, York University , 4700 Keele Street, 126 BSB, Toronto , Ontario, Canada M3J 1P3 The Cognitive Reflection Test (CRT; Frederick, 2005) is designed to measure the tendency to override a prepotent response alternative that is incorrect and to engage in further reflection that leads to the correct response. In this study, we showed that the CRT is a more potent predictor of performance on a wide sample of tasks from the heuristics-and-biases literature than measures of cognitive ability, thinking dispositions, and executive functioning. Although the CRT has a substantial correlation with cognitive ability, a series of regression analyses indicated that the CRT was a unique predictor of performance on heuristics-and-biases tasks. It accounted for substantial additional variance after the other measures of individual differences had been statistically controlled. We conjecture that this is because neither intelligence tests nor measures of executive functioning assess the tendency toward miserly processing in the way that the CRT does. We argue that the CRT is a particularly potent measure of the tendency toward miserly processing because it is a performance measure rather than a self-report measure. - The Cognitive Reflection Test (CRT) is a three-item measure introduced into the journal literature by Frederick (2005). The task is designed to measure the tendency to override a prepotent response alternative that is incorrect and to engage in further reflection that leads to the correct response. The quintessential item from the CRT was first discussed by Kahneman and Frederick (2002) in an article that reframed the heuristics-and-biases literature in terms of the concept of attribute substitution. The problem is as follows: A bat and a ball cost $1.10 in total. The bat costs $1 more than the ball. How much does the ball cost? When they answer this problem, many people show a characteristic that is common to many reasoning errors: They behave like cognitive misers (Dawes, 1976; Simon, 1955, 1956; Stanovich, 2009b; Taylor, 1981; Tversky & Kahneman, 1974). They give the first response that comes to mind10 centswithout thinking further and realizing that this cannot be right. The bat would then have to cost $1.10, and the total cost would then be $1.20 rather than the required $1.10. People often do not think deeply enough to realize their error, and cognitive ability is no guarantee against making the error. Frederick (2005) found that large numbers of highly select university students at MIT, Princeton, and Harvard were cognitive misers; they responded that the cost was 10 cents, rather than the correct answer. .. 5 cents. This problem and the two others (see the Method section below) on the CRT seem at first glance to be similar to the well-known insight problems in the problem-solving literature, but they in fact display a critical difference. Classic insight problems (see Gilhooly & Fioratou, 2009; Gilhooly & Murphy, 2005) do not usually trigger an attractive alternative response. Instead, the participant sits lost in thought trying to reframe the problem correctlyas in, for example, the classic nine-dot problem. The three problems on the CRT are of interest to researchers working in the heuristics-and-biases tradition because a strong alternative response is initially primed and then must be overridden. As Kahneman and Frederick made clear in their 2002 paper, this framework of an incorrectly primed initial response that must be overridden fits in nicely with currently popular dual-process frameworks (De Neys & Glumicic, 2008; Evans, 1984, 2008, 2010; Evans & Frankish, 2009; Lieberman, 2007, 2009; Sloman, 1996, 2002; Stanovich, 1999, 2009a, 2011). Kahneman (2000) pointed out that such a framework had been an underlying assumption of his earlier work with Tversky. The CRT would seem to be ideally constructed as a predictor of performance on heuristics-and-biases tasks, but the data have been inconsistent. Frederick (2005) observed that with as few as three items, his CRT could predict performance on measures of temporal discounting, the tendency to choose high-expected-value gambles, and framing effects. Likewise, Cokely and Kelley (2009) found a correlation of .27 between performance on the CRT and the proportion of choices consistent with expected value. In contrast, Campitelli and Labollita (2010) found little relation between CRT performance and the choice of high-expected-value gambles. Oechssler, Roider, and Schmitz (2009) found the CRT to be related to the number of expected-value choices and the tendency to commit the conjunction fallacy. In contrast, Obrecht, Chapman, and Gelman (2009) found no relation between CRT performance and the degree of encounter frequency bias. Finally, Koehler and James (2010) found significant correlations between the CRT and the use of and endorsement of maximizing strategies on probabilistic prediction tasks. In the present article, we explore the predictive properties of the CRT in a much wider range of the heuristics-and-biases tasks. Additionally, however, we attempt to uncover some of the underlying psychological structure of the CRT. This is necessary because on the surface, the CRT appears to be a somewhat complex measure. It seems to carry properties across the boundary of an important distinction in classical personality and psychometric workthat is, the distinction between cognitive abilities and thinking dispositions. This conceptual distinction follows from differentiating optimal (sometimes termed maximal) performance situations and typical performance situations (see Ackerman, 1994, 1996; Ackerman & Heggestad, 1997; Ackerman & Kanfer, 2004; see also Cronbach, 1949; Matthews, Zeidner, & Roberts, 2002). Typical performance situations are unconstrained, in that no overt instructions to maximize performance are given, and the task interpretation is determined to some extent by the participant. The goals to be pursued in the task are left somewhat open. The issue is what a person would typically do in such a situation, given few constraints (see Stanovich, 2009b). In contrast, optimal performance situations are those in which the task interpretation is determined externally (not left to the participant). The person performing the task is instructed to maximize performance. Duckworth (2009) has discussed the surprisingly weak relation between typical and maximal performance across a variety of domains. For example, Sackett, Zedeck, and Fogli (1988) found that there were very low correlations between the maximal item-processing efficiency that supermarket cashiers could attain and the typical processing efficiency that t (...truncated)