SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods

Ribeiro et al. EPJ Data Science (2016) 5:23 DOI 10.1140/epjds/s13688-016-0085-1 REGULAR ARTICLE Open Access SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods Filipe N Ribeiro1,2* , Matheus Araújo1 , Pollyanna Gonçalves1 , Marcos André Gonçalves1 and Fabrício Benevenuto1 * Correspondence: fi[email protected] 1 Computer Science Department, Federal University of Minas Gerais, Belo Horizonte, Brazil 2 Computer and Systems Department, Federal University of Ouro Preto, Joao Monlevade, Brazil Abstract In the last few years thousands of scientific papers have investigated sentiment analysis, several startups that measure opinions on real data have emerged and a number of innovative products related to this theme have been developed. There are multiple methods for measuring sentiments, including lexical-based and supervised machine learning methods. Despite the vast interest on the theme and wide popularity of some methods, it is unclear which one is better for identifying the polarity (i.e., positive or negative) of a message. Accordingly, there is a strong need to conduct a thorough apple-to-apple comparison of sentiment analysis methods, as they are used in practice, across multiple datasets originated from different data sources. Such a comparison is key for understanding the potential limitations, advantages, and disadvantages of popular methods. This article aims at filling this gap by presenting a benchmark comparison of twenty-four popular sentiment analysis methods (which we call the state-of-the-practice methods). Our evaluation is based on a benchmark of eighteen labeled datasets, covering messages posted on social networks, movie and product reviews, as well as opinions and comments in news articles. Our results highlight the extent to which the prediction performance of these methods varies considerably across datasets. Aiming at boosting the development of this research area, we open the methods’ codes and datasets used in this article, deploying them in a benchmark system, which provides an open API for accessing and comparing sentence-level sentiment analysis methods. Keywords: sentiment analysis; benchmark; methods evaluation 1 Introduction Sentiment analysis has become an extremely popular tool, applied in several analytical domains, especially on the Web and social media. To illustrate the growth of interest in the field, Figure  shows the steady growth on the number of searches on the topic, according to Google Trends,a mainly after the popularization of online social networks (OSNs). More than , articles have been written about sentiment analysis and various startups are developing tools and strategies to extract sentiments from text []. The number of possible applications of such a technique is also considerable. Many of them are focused on monitoring the reputation or opinion of a company or a brand with © 2016 Ribeiro et al. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Ribeiro et al. EPJ Data Science (2016) 5:23 Page 2 of 29 Figure 1 Searches on Google for the Query: ‘Sentiment Analysis’. This figure shows the steady growth on the number of searches on the topic, according to Google Trends, mainly after the popularization of online social networks (OSNs). the analysis of reviews of consumer products or services []. Sentiment analysis can also provide analytical perspectives for financial investors who want to discover and respond to market opinions [, ]. Another important set of applications is in politics, where marketing campaigns are interested in tracking sentiments expressed by voters associated with candidates []. Due to the enormous interest and applicability, there has been a corresponding increase in the number of proposed sentiment analysis methods in the last years. The proposed methods rely on many different techniques from different computer science fields. Some of them employ machine learning methods that often rely on supervised classification approaches, requiring labeled data to train classifiers []. Others are lexical-based methods that make use of predefined lists of words, in which each word is associated with a specific sentiment. The lexical methods vary according to the context in which they were created. For instance, LIWC [] was originally proposed to analyze sentiment patterns in formally written English texts, whereas PANAS-t [] and POMS-ex [] were proposed as psychometric scales adapted to the Web context. Overall, the above techniques are acceptable by the research community and it is common to see concurrent important papers, sometimes published in the same computer science conference, using completely different methods. For example, the famous Facebook experiment [] which manipulated users feeds to study emotional contagion, used LIWC []. Concurrently, Reis et al. used SentiStrength [] to measure the negativeness or positiveness of online news headlines [, ], whereas Tamersoy [] explored VADER’s lexicon [] to study patterns of smoking and drinking abstinence in social media. As the state-of-the-art has not been clearly established, researchers tend to accept any popular method as a valid methodology to measure sentiments. However, little is known about the relative performance of the several existing sentiment analysis methods. In fact, most of the newly proposed methods are rarely compared with all other pre-existing ones using a large number of existing datasets. This is a very unusual situation from a scientific perspective, in which benchmark comparisons are the rule. In fact, most applications and experiments reported in the literature make use of previously developed methods exactly how they were released with no changes and adaptations and with none or almost none parameter setting. In other words, the methods have been used as a black-box, without a deeper investigation on their suitability to a particular context or application. To sum up, existing methods have been widely deployed for developing applications without a deeper understanding regarding their applicability in different contexts or their Ribeiro et al. EPJ Data Science (2016) 5:23 Page 3 of 29 advantages, disadvantages, and limitations in comparison with each another. Thus, there is a strong need to conduct a thorough apple-to-apple comparison of sentiment analysis methods, as they are used in practice, across multiple datasets originated from different data sources. This state-of-the-practice situation is what we propose to investigate in this article. We do this by providing a thorough benchmark comparison of twenty-four state-of-thepractice (...truncated)