A population-based investigation of participation rate and self-selection bias in momentary data capture and survey studies

Current Psychology https://doi.org/10.1007/s12144-023-04426-2 A population‑based investigation of participation rate and self‑selection bias in momentary data capture and survey studies Arthur A. Stone1,2 · Stefan Schneider1,2 · Joshua M. Smyth3 · Doerte U. Junghaenel1,2 · Mick P. Couper4 · Cheng Wen2 · Marilyn Mendez2 · Sarah Velasco2 · Sarah Goldstein2 Accepted: 16 February 2023 © The Author(s) 2023 Abstract Participant selection bias is of concern to researchers conducting surveys of all types. For momentary data capture studies, such as Ecological Momentary Assessment, the level of burden associated with these techniques and the possibility of low uptake rates makes the concerns especially salient. This study invited 3,000 individuals to participate in a study of health and mood and recorded the uptake rates at various points in the process. Respondents expressing interest in participating in general were randomized into a one-time survey, a low-burden momentary study, or a high-burden momentary study. Overall, 85.9% of the sample did not respond to the study invitation (including confirming non-interest); 6.9% of the sample expressed interest in the study by completing a brief survey; 2.1% agreed to participate in the study when the protocol specifics were described (none of the study protocols were actually run). Whites were more likely to complete the survey. Of those completing the survey, individuals who reported higher income, a more “open” personality, better typing skills, better computer skills, who viewed the research topic as important, and who expressed interest in research on daily feelings more likely consented to being enrolled in the experiment. The number of prior surveys taken had an inverted-U shaped association with participation in this study. Finally, all individuals randomized to the one-time survey group agreed to participate compared to two-thirds of individuals in the momentary groups. These results suggest that participant selection bias may affect both one-time survey and momentary data capture studies, with the caveat that the degree of such bias will be related to a study’s hypotheses. Keywords EMA · Survey · Selection bias · Participation rate · General population This paper concerns self-selection bias in survey research generally and particularly with studies using momentary assessment strategies, which are often referred to as Ecological Momentary Assessment (EMA, the term that will be used throughout the paper (Shiffman et al., 2008; Stone & Shiffman, 1994), the Experience Sampling Method (ESM) (Conner et al., 2009; Csikszentmihalyi & Hunter, 2003), * Arthur A. Stone 1 Department of Psychology, University of Southern California, Los Angeles, CA, USA 2 Dornsife Center for Self‑Report Science, University of Southern California, Los Angeles, CA, USA 3 Department of Biobehavioral Health, The Pennsylvania State University, State College, PA, USA 4 Institute for Social Research, University of Michigan, Ann Arbor, MI, USA and Ambulatory Assessment (Ebner-Priemer & Trull, 2009; Wright & Zimmermann, 2019). Momentary studies are often based on a relatively small number of study participants and there may be concerns about making inferences from these studies to broader populations. As the EMA field matures, it is important to examine the feasibility of recruiting individuals from the general population – the focus of this paper – or from more specialized populations (e.g., patients) given our perception that many momentary researchers believe that only a small proportion of individuals approached for a momentary study actually participate (Stone et al., in press). The specific concern that this paper evaluates is that people with certain characteristics will not agree to participate; if so, this creates a threat to the external validity (Cook & Campbell, 1979) of the results. The methodological phenomenon described above is long recognized and usually referred to as self-selection bias. It has received considerable attention in the social sciences and 13 Vol.:(0123456789) Current Psychology the survey research literature (Bethlehem, 2010; Heckman, 2010). A number of studies have focused on motivational barriers of uptake into studies (in general, not focused on EMA) by examining the relevance of study content for prospective participants (Materia & Smyth, 2021), the availability of data collection devices (Jäckle et al., 2019), and by examining appropriate monetary and non-monetary incentives (Yu & Cooper, 1983). There has been less attention to self-selection bias in the field of momentary data capture and virtually no research in general populations on the topic (Gabriel et al., 2019; Hektner et al., 2007; Scollon et al., 2009). A recent review of “pressing” issues for the field of EMA has identified participant self-selection as a major concern (Stone et al., 2023) and suggested possible ways of exploring the topic. The relevance of self-selection bias, defined as individuals deciding for themselves whether or not to participate in a study, lies in the possibility that the sample will not adequately represent the population from which it was drawn. That is, that those declining participation in an EMA study will be different in some ways from those agreeing to participate. Under the assumption of sound sampling strategies and excellent uptake rates, studies achieving high uptake reduce the threat of self-selection bias, whereas the threat likely increases with lower uptake rates. However, the definition of what constitutes “good” uptake rates is not a fixed value: it depends upon the associations being studied and how selection impacts the relevant variables. It is also the case that less than perfect uptake rates do not necessarily result in self-selection processes that will bias the external validity of the results, because such bias occurs when the characteristics of those not participating (measured or not measured) impact the associations under consideration. For example, imagine an EMA study concerned with understanding whether momentary pain is associated with a greater likelihood of momentary social withdrawal using data from random prompts. Further, let’s assume that this relationship is moderated by trait extroversion such that pain relates to social withdrawal among introverts much more than among extroverts and that extroverts are more likely to participate in an EMA study. A study of this topic would be therefore likely to under-estimate (bias) the observed effect of momentary pain on social withdrawal, because extroverts were more likely to participate and because the personality characteristic is associated with the effect being studied. However, if the goal of the study was to examine the relationship between momentary pain and medication taking and this association was not related to extroversion, then the results would not be biased. Thus, these points need to be considered in the evaluation of self-se (...truncated)