Are markets more accurate than polls? The surprising informational value of “just asking”

Judgment and Decision Making, Vol. 14, No. 2, March 2019, pp. 135–147 Are markets more accurate than polls? The surprising informational value of “just asking” Jason Dana∗ Pavel Atanasov† Philip Tetlock† Barbara Mellers† Abstract Psychologists typically measure beliefs and preferences using self-reports, whereas economists are much more likely to infer them from behavior. Prediction markets appear to be a victory for the economic approach, having yielded more accurate probability estimates than opinion polls or experts for a wide variety of events, all without ever asking for self-reported beliefs. We conduct the most direct comparison to date of prediction markets to simple self-reports using a within-subject design. Our participants traded on the likelihood of geopolitical events. Each time they placed a trade, they first had to report their belief that the event would occur on a 0–100 scale. When previously validated aggregation algorithms were applied to self-reported beliefs, they were at least as accurate as prediction-market prices in predicting a wide range of geopolitical events. Furthermore, the combination of approaches was significantly more accurate than prediction-market prices alone, indicating that self-reports contained information that the market did not efficiently aggregate. Combining measurement techniques across behavioral and social sciences may have greater benefits than previously thought. Keywords: prediction, forecast, judgment, prediction markets, self-reports, surveys 1 Introduction Behavioral and social scientists have long disagreed over how best to measure mental states. While psychologists clearly value behavioral measures, they quite often measure beliefs and preferences by simply asking people to self-report them on a numerical scale. And while economists place value on people’s judgments, they tend to place greater value on inferring preferences and beliefs from behavior. For example, if a person claims that the United States is on the verge of an economic collapse or that a climate disaster is imminent, an economist might look at that person’s investment portfolio or disaster preparedness to reveal whether that person All raw data are publicly available via the Open Science Framework and can be accessed at https://dataverse.harvard.edu/dataverse/gjp. B. Mellers, J. Dana, P. Atanasov and P. Tetlock developed the study concept. J. Dana and P. Atanasov developed relevant tests and performed all data analyses. J. Dana, P. Atanasov, and B. Mellers wrote the manuscript with critical input from P. Tetlock. All authors approved the final version of the manuscript for submission. We thank Phillip Rescober for data assistance and Paul Tetlock and Joi Ito for helpful discussions. This research was supported by the Intelligence Advanced Research Projects Activity (IARPA) via the Department of Interior National Business Center (DoI/NBC) Contract No. D11PC20061. The U.S. government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation thereon. The views and conclusions expressed herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/ NBC, or the U.S. government. Copyright: © 2019. The authors license this article under the terms of the Creative Commons Attribution 3.0 License. ∗ Yale University. Email: . † Department of Psychology, University of Pennsylvania really believes these statements. Indeed, the unwillingness of economists to rely on survey questions has been called an important divide with other behavioral scientists (Bertrand & Mullainathan, 2001). Perhaps the most impressive demonstration of the power of using revealed beliefs is the resounding success of prediction markets. Prediction markets create contracts that pay a fixed amount if an event occurs, and then allow people to trade on the contract by submitting buying or selling prices in a manner similar to the stock market. The price at which the contract trades at a given time can be taken to be the market’s collective probability estimate of the event occurring. For example, suppose the event to be predicted was the winner of the 2016 US presidential election, and that a contract paid $100 if Hillary Clinton won. If the contract last traded at $60 – that is, someone just purchased the contract from someone else for $60 — one could use that price as a likelihood prediction of Clinton winning of 60%. In other words, if a risk-neutral market is valuing a risky $100 contract at $60, it implies that the expected value of the contract is $60 and thus that it will pay out with probability .6. Using prediction market prices in this manner has yielded impressively accurate predictions for a wide array of outcomes, such as the winners of elections or sporting events, typically exceeding the accuracy of “just asking” methods such as opinion polls or expert forecasts (see Wolfers & Zitzewitz, 2004; Ray, 2006, for reviews). When market participants have some intrinsic interest in trying to predict results, even markets with modest incentives or no incentives have been shown to be effective. As examples, small markets using academics as participants predict which behavioral sci- 135 Judgment and Decision Making, Vol. 14, No. 2, March 2019 ence experiments will successfully replicate (e.g., Camerer et al., 2018) and “play money” markets in which participants play for prestige can be as accurate as real-money markets (Pennock et al, 2001; Servan-Schreiber et al., 2004). Because these probability forecasts are obtained without ever asking anyone to self-report their beliefs, the success of prediction markets appears to be a victory of the economic approach and a repudiation of relying on self-reports. The classic explanation for why prediction markets are so successful is that they are efficient mechanisms for integrating information useful to making predictions. To see why, suppose that someone had information that suggested an event was much more likely to occur than the current market price suggested. That person would now have the incentive to buy the contract because its expected value would greatly exceed its cost. The balance of such beliefs would eventually push the price up. Others might have pieces of information that suggest the event is unlikely, motivating them to sell and putting downward pressure on the price. In the end, the market price will tend to reflect the balance of information that participants have. Indeed, when traders try to engage in market manipulation, buying and selling with the intent of changing the price to provide misinformation, their attempts usually fail and the market can become even more informative due to the incentives for traders to act on their true beliefs (Hanson, Oprea & Porter, 2006). In theory, there should be no information in their self-reports that is not al (...truncated)