Bandit strategies in social search: the case of the DARPA red balloon challenge
Chen et al. EPJ Data Science
Bandit strategies in social search: the case of the DARPA red balloon challenge
Haohui Chen 1 2
Iyad Rahwan 0
Manuel Cebrian 2
0 The Media Laboratory, Massachusetts Institute of Technology , Cambridge, Massachusetts , USA
1 Faculty of Information Technology, Monash University , Caulfield, Victoria , Australia
2 Data61 Unit, Commonwealth Scientific and Industrial Research Organization , Melbourne, Victoria , Australia
Collective search for people and information has tremendously benefited from emerging communication technologies that leverage the wisdom of the crowds, and has been increasingly influential in solving time-critical tasks such as the DARPA Network Challenge (DNC, also known as the Red Balloon Challenge). However, while collective search often invests significant resources in encouraging the crowd to contribute new information, the effort invested in verifying this information is comparable, yet often neglected in crowdsourcing models. This paper studies how the exploration-verification trade-off displayed by the teams modulated their success in the DNC, as teams had limited human resources that they had to divide between recruitment (exploration) and verification (exploitation). Our analysis suggests that team performance in the DNC can be modelled as a modified multi-armed bandit (MAB) problem, where information arrives to the team originating from sources of different levels of veracity that need to be assessed in real time. We use these insights to build a data-driven agent-based model, based on the DNC's data, to simulate team performance. The simulation results match the observed teams' behavior and demonstrate how to achieve the best balance between exploration and exploitation for general time-critical collective search tasks.
crowdsourcing; exploration; exploitation; misinformation; disinformation; social search; bandit problem
1 Introduction
Crowdsourcing, the use of the Internet to solicit contributions from large groups of
people, has been shown to be very effective in time-critical tasks, ranging from manhunts
[–], to influenza detection [], to crisis-mapping [, , ]. However, time-critical
crowdsourcing tasks often reward the collection of new information, but ignore the efforts of
verification. Crowds tend to explore new information but seldom verify it autonomously,
and exploration effort often dominates. This causes information overload, where
misinformation (caused by error) and disinformation (caused by deliberate malice) conceal true
information [], posing a significant challenge to crowdsourcing. In the context of disaster
response, while online social media is a highly-effective crowdsourcing tool, it also makes
it nearly costless to spread false information []. Misinformation has impeded search and
rescue operations [], and sometimes it can go as far as harming innocent people. For
example, during the manhunt for the Boston Marathon bombers, the crowd wrongly
identified one missing student, Sunil Tripathi, as a suspect. It subsequently emerged that he
© 2016 Chen et al. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,
provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and
indicate if changes were made.
had died days before the bombings, yet misinformation was spread in , tweets [].
Scholars have identified this problem and paid attention to the detection of false
information. Gupta et al. [] and Boididou et al. [] building up on Canini et al. []’s work, use
contents and corresponding authors’ profiles to classify false information, and achieve
relatively high accuracy. However, in reality, information arrives from various channels, e.g.
phone calls, text messages or online social media. Therefore, there is no universal method
of processing the information and even classifying it in a short period of time. This paper
does not attempt to build a classifier or a universal strategy for discriminating
misinformation or disinformation from correct entries. Rather, we assume that, based on the
discussion above, the success of a time-critical task requires not just exploring new information
(exploration) but also verification (exploitation). Given that an individual or organization
has limited resources, exploration and exploitation are regarded as two competing
processes []. Therefore, this paper explores how to balance exploration and exploitation in
time-critical crowdsourcing tasks.
We use DARPA Network Challenge (DNC) as the study case. In , DARPA launched
a competition, which aims to evaluate the power of social networks and media in
mobilizing crowds. Ten red weather balloons were placed at undisclosed locations throughout
the United States. Participating teams (...truncated)