Sensible reasoning in two tasks: Rule discovery and hypothesis evaluation

HILARY H. FARRIS 0 1 RUSSELL REVLIN 0 1 0 ment, University of California , Santa Barbara, CA 93106 1 University of California , Santa Barbara, California The hypothesis testing skills of undergraduates were measured in two tasks: the 2-4-6 rule discovery task in which students generate and assess hypotheses, and a hypothesis evaluation task, which requires only the assessment of hypotheses. The results of Experiments 1 and 2 show that the students consistently employed a disconfirmation strategy when assessing hypotheses, but employed a counterfactual inference strategy when they also were required to generate the hypotheses. The results of Experiment 3 suggest that the selection of the hypothesis testing strategy reflected a balance between the logical requirements of the task and the desirability of possible outcomes. Taken together, the findings support a more consistent picture of human rationality across tasks, and suggest alternatives to accounts of confirmation bias. - 2,4,6) to begin the task and were told that the triple con formed to the rule. They were asked to evaluate their hypotheses by generating triples of numbers. The ex perimenter would provide feedback on whether the tri ple was consistent with the rule in question. This task is analogous to one faced by scientists, with the seed triple functioning as an initiating observation, and the act of generating the triple is equivalent to performing an ex periment. The methodology is an extension of the work of Bruner, Goodnow, and Austin (1956). While there are a number of ways in which the critical evaluation of hypotheses can be carried out, Wason fo cused primarily on the disconfirmation strategy in which the hypothesis tester generates counterexamples of the hypotheses under consideration. The formal basis for the strategy is as follows: (I) If hypothesis HI is true, then triple T1 is true and 1'2 is false; and (2) if 1'2 is true, then HI is false. However, if the reasoner follows a con firmation strategy, then a third rule would be in effect: (3) If T1 is true, then HI is true. Of course, this is a log ically unsound rule, because the most you can conclude from observing that T1 is true is that HI is plausible. The use of disconfirmation is operationalized in Ta ble 1. Students following this rule of inference should seek to "falsify" their hypothesis by generating triples that would be false if their hypothesis is true (as with rule 2 above). This is in keeping with the philosophical notion of falsifiability and the critical testing of hypotheses (e.g., Popper, 1972). Studies examining adult reasoning in Wason's 2-4-6 rule discovery task have been concerned primarily with extensions of the discovery paradigm in various environ ments (Gorman, Gorman, Latta, & Cunningham, 1984; Gorman, Stafford, & Gorman, 1987; Mahoney & DeMonbreun, 1977; Mynatt, Doherty, & Tweney, 1977, 1978). Other studies have been directed to facilitate stu dents' use of disconfirmation within the Popperian frame work of falsification (e.g., Gorman, 1986; Gorman & Gorman, 1984; Tukey, 1986; Tweney et al., 1980; Wetherick, 1962). In general, these studies concur with Wason's analysis: A majority of individuals fail to con sider alternate hypotheses and, instead, seek to confirm, by enumeration, a favorite hypothesis. This phenomenon in rule discovery is referred to as confirmation bias. It should be noted that the correct solution does not ab solutely require that the disconfirmation strategy be fol lowed (Tukey, 1986; Tweney et al., 1980). For exam ple, there are other rational strategies that one may em ploy in rule discovery other than disconfirmation (see a Bayesian analysis by Baron, 1985; the positive test strategy of Klayman & Ha, 1987; as well as the scho larly analyses by Hardin, 1980, and Tukey, 1986). We present evidence here to suggest the presence of a specific rational strategy, counterfactual reasoning, that is part of the hypothesis testing process (Rescher, 1964; Revlis & Hayes, 1972). When we follow this procedure, we consider the truth of a hypothesis by assuming it is false and observing the results. There are many derived forms of such reasoning, including reductio ad absurdum arguments in which one assumes that the antithesis of a to-be-proven argument is correct and evaluates the con sequences (see, also, applications of counterfactual infer ence to language processing by Hornby, 1974). The steps a student using this strategy might follow in the rule discovery task are shown in Table 1 and are il lustrated by the following example. Suppose we believe that the rule the experimenter has in mind is "even num bers; " this is tested by assuming that the hypothesized rule is false and positing, for example, that "odd num bers" is true. We then generate a triple that is consistent with the latter hypothesis (e.g., 3, 5, 7). If the ex perimenter confirms that the triple is consistent with the experimenter's rule, then "even numbers" is indeed false. If the triple is inconsistent with the rule, then "odd num bers" is false and "even numbers" is plausible. Notice that while the triple generated by the hypothesis tester may confirm the stated hypothesis, it actually disconfirms the target hypothesis that is assumed to be false. Unfor tunately, if the reasoner follows the counterfactual Disconfinnation Assume: Hypothesis A is correct Generate: Prediction (triple) that is inconsistent with A Evaluate Feedback: (a) If "Yes", then assumption about A is incorrect (b) If "No", then A is likely to be correct strategy, the substantial frequency of confirming triples can be misinterpreted by the experimenter as evidence for a bias to confirm. In the present study, we examine the possibility that stu dents may engage in such forms of counterfactual infer ence during the rule discovery task and that the resulting behaviors may appear to be confirmation, when in fact they reflect sensible hypothesis testing strategies. To evaluate the possibility that reasoners employ the counterfactual strategy, we must depart somewhat from the traditional paradigm. The statistical analyses in previ ous studies of rule discovery have tested whether the student-generated instances were consistent with previ ous hypotheses. In contrast, we are concerned with whether the hypotheses themselves are compatible with prior hypotheses. This approach gives us a more com plete picture of the entire hypothesis testing strategy, be cause it provides an opportunity to distinguish between students using disconfirmation and those using counter factual inference at the local level of a single statement of a hypothesis with its companion triple. HYPOTHESIS EVALUATION Decisions on hypothesis evaluation tasks provide a more optimistic picture of reasoning strategies. Tschirgi (1980) presented students with vignettes in which a story charac ter had to test a hypothesis about the importance of one of three variables for an event in the story (see Appen dix (...truncated)