Bandit strategies in social search: the case of the DARPA red balloon challenge

EPJ Data Science, Jun 2016

Collective search for people and information has tremendously benefited from emerging communication technologies that leverage the wisdom of the crowds, and has been increasingly influential in solving time-critical tasks such as the DARPA Network Challenge (DNC, also known as the Red Balloon Challenge). However, while collective search often invests significant resources in encouraging the crowd to contribute new information, the effort invested in verifying this information is comparable, yet often neglected in crowdsourcing models. This paper studies how the exploration-verification trade-off displayed by the teams modulated their success in the DNC, as teams had limited human resources that they had to divide between recruitment (exploration) and verification (exploitation). Our analysis suggests that team performance in the DNC can be modelled as a modified multi-armed bandit (MAB) problem, where information arrives to the team originating from sources of different levels of veracity that need to be assessed in real time. We use these insights to build a data-driven agent-based model, based on the DNC’s data, to simulate team performance. The simulation results match the observed teams’ behavior and demonstrate how to achieve the best balance between exploration and exploitation for general time-critical collective search tasks.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://link.springer.com/content/pdf/10.1140%2Fepjds%2Fs13688-016-0082-4.pdf

Bandit strategies in social search: the case of the DARPA red balloon challenge

Chen et al. EPJ Data Science Bandit strategies in social search: the case of the DARPA red balloon challenge Haohui Chen 1 2 Iyad Rahwan 0 Manuel Cebrian 2 0 The Media Laboratory, Massachusetts Institute of Technology , Cambridge, Massachusetts , USA 1 Faculty of Information Technology, Monash University , Caulfield, Victoria , Australia 2 Data61 Unit, Commonwealth Scientific and Industrial Research Organization , Melbourne, Victoria , Australia Collective search for people and information has tremendously benefited from emerging communication technologies that leverage the wisdom of the crowds, and has been increasingly influential in solving time-critical tasks such as the DARPA Network Challenge (DNC, also known as the Red Balloon Challenge). However, while collective search often invests significant resources in encouraging the crowd to contribute new information, the effort invested in verifying this information is comparable, yet often neglected in crowdsourcing models. This paper studies how the exploration-verification trade-off displayed by the teams modulated their success in the DNC, as teams had limited human resources that they had to divide between recruitment (exploration) and verification (exploitation). Our analysis suggests that team performance in the DNC can be modelled as a modified multi-armed bandit (MAB) problem, where information arrives to the team originating from sources of different levels of veracity that need to be assessed in real time. We use these insights to build a data-driven agent-based model, based on the DNC's data, to simulate team performance. The simulation results match the observed teams' behavior and demonstrate how to achieve the best balance between exploration and exploitation for general time-critical collective search tasks. crowdsourcing; exploration; exploitation; misinformation; disinformation; social search; bandit problem 1 Introduction Crowdsourcing, the use of the Internet to solicit contributions from large groups of people, has been shown to be very effective in time-critical tasks, ranging from manhunts [–], to influenza detection [], to crisis-mapping [, , ]. However, time-critical crowdsourcing tasks often reward the collection of new information, but ignore the efforts of verification. Crowds tend to explore new information but seldom verify it autonomously, and exploration effort often dominates. This causes information overload, where misinformation (caused by error) and disinformation (caused by deliberate malice) conceal true information [], posing a significant challenge to crowdsourcing. In the context of disaster response, while online social media is a highly-effective crowdsourcing tool, it also makes it nearly costless to spread false information []. Misinformation has impeded search and rescue operations [], and sometimes it can go as far as harming innocent people. For example, during the manhunt for the Boston Marathon bombers, the crowd wrongly identified one missing student, Sunil Tripathi, as a suspect. It subsequently emerged that he © 2016 Chen et al. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. had died days before the bombings, yet misinformation was spread in , tweets []. Scholars have identified this problem and paid attention to the detection of false information. Gupta et al. [] and Boididou et al. [] building up on Canini et al. []’s work, use contents and corresponding authors’ profiles to classify false information, and achieve relatively high accuracy. However, in reality, information arrives from various channels, e.g. phone calls, text messages or online social media. Therefore, there is no universal method of processing the information and even classifying it in a short period of time. This paper does not attempt to build a classifier or a universal strategy for discriminating misinformation or disinformation from correct entries. Rather, we assume that, based on the discussion above, the success of a time-critical task requires not just exploring new information (exploration) but also verification (exploitation). Given that an individual or organization has limited resources, exploration and exploitation are regarded as two competing processes []. Therefore, this paper explores how to balance exploration and exploitation in time-critical crowdsourcing tasks. We use DARPA Network Challenge (DNC) as the study case. In , DARPA launched a competition, which aims to evaluate the power of social networks and media in mobilizing crowds. Ten red weather balloons were placed at undisclosed locations throughout the United States. Participating teams (...truncated)


This is a preview of a remote PDF: https://link.springer.com/content/pdf/10.1140%2Fepjds%2Fs13688-016-0082-4.pdf

Haohui Chen, Iyad Rahwan, Manuel Cebrian. Bandit strategies in social search: the case of the DARPA red balloon challenge, EPJ Data Science, 2016, pp. 20, Volume 5, Issue 1, DOI: 10.1140/epjds/s13688-016-0082-4