Online psychophysics: reaction time effects in cognitive experiments

Behavior Research Methods, Aug 2016

Using the Internet to acquire behavioral data is currently on the rise. However, very basic questions regarding the feasibility of online psychophysics are still open. Here, we aimed to replicate five well-known paradigms in experimental psychology (Stroop, Flanker, visual search, masked priming, attentional blink) in three settings (classical “lab”, “web-in-lab”, “web”) to account for possible changes in technology and environment. Lab and web-in-lab data were both acquired in an in-lab setting with lab using “Gold Standard” methods, while web-in-lab used web technology. This allowed for a direct comparison of potential differences in acquisition software. To account for additional environmental differences, the web technology experiments were published online to participate from home (setting web), thereby keeping the software and experimental design identical and only changing the environmental setting. Our main results are: First, we found an expected fixed additive timing offset when using web technology (M = 37 ms, SD = 8.14) and recording online (M = 87 ms, SD = 16.04) in comparison to lab data. Second, all task-specific effects were reproduced except for the priming paradigm, which couldn’t be replicated in any setting. Third, there were no differences in error rates, which are independent of the timing offset. This finding further supports the assumption of data equality over all settings. Fourth, we found that browser type might be influencing absolute reaction times. Together, these results contribute to the slowly but steadily growing literature that online psychophysics is a suitable complement – or even substitute – to lab data acquisition.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://link.springer.com/content/pdf/10.3758%2Fs13428-016-0783-4.pdf

Online psychophysics: reaction time effects in cognitive experiments

Behav Res Online psychophysics: reaction time effects in cognitive experiments Kilian Semmelmann 0 Sarah Weigelt 0 0 Developmental Neuropsychology, Department of Psychology, Ruhr-University Bochum , Universitätsstr. 150, 44801 Bochum , Germany 1 Kilian Semmelmann Using the Internet to acquire behavioral data is currently on the rise. However, very basic questions regarding the feasibility of online psychophysics are still open. Here, we aimed to replicate five well-known paradigms in experimental psychology (Stroop, Flanker, visual search, masked priming, attentional blink) in three settings (classical Blab^, Bweb-inlab^, Bweb^) to account for possible changes in technology and environment. Lab and web-in-lab data were both acquired in an in-lab setting with lab using BGold Standard^ methods, while web-in-lab used web technology. This allowed for a direct comparison of potential differences in acquisition software. To account for additional environmental differences, the web technology experiments were published online to participate from home (setting web), thereby keeping the software and experimental design identical and only changing the environmental setting. Our main results are: First, we found an expected fixed additive timing offset when using web technology (M = 37 ms, SD = 8.14) and recording online (M = 87 ms, SD = 16.04) in comparison to lab data. Second, all taskspecific effects were reproduced except for the priming paradigm, which couldn't be replicated in any setting. Third, there were no differences in error rates, which are independent of the timing offset. This finding further supports the assumption of data equality over all settings. Fourth, we found that browser type might be influencing absolute reaction times. Together, these results contribute to the slowly but steadily growing Psychophysics; Web technology; Online study; Replication; Cognitive psychology; Reaction time - literature that online psychophysics is a suitable complement – or even substitute – to lab data acquisition. The Internet has a huge impact on every aspect of today’s life. In experimental psychology, the introduction and establishment of online surveys has allowed researchers to reach more people from a broader background more easily, efficiently, and faster than ever before (Reips, 2000) . Thanks to the higher heterogeneity of the participants (age, gender, origin, social status; see Birnbaum, 2004) online studies allow us to tackle one of the biggest open questions in psychology: To what extent can the results of a single study be mapped to the whole population? Furthermore, through the worldwide availability of online experiments, scientists have easier access to hard-toreach populations, benefit from a double-blind situation, and are able to conduct parallel experiments independent of equipment or available experimenters (Gosling, Vazire, Srivastava, & John, 2000; Reips, 2000) . For example Joinson (2001) found through the results of three studies that participants in an anonymous, computer-mediated research environment disclose significantly more information than in classical research settings. A work by Reimers (2007) collected data from over 255,000 participants in a study about sex differences in Britain – one of the largest up to date. Collecting such large samples will help scientists to battle the issue of insufficient power, which often renders studies irreplicable. A much more selective, but still numerous, sample was collected by Cohen, Collins, Darkes, and Gwartney (2007), who used online message boards to get nearly 2,000 responses from a very specific population of non-medical anabolic steroid users in the USA. These and more advantages have been covered in excessive exploration and validation studies (Birnbaum, 2000; Gosling et al., 2000; Skitka & Sargis, 2006; amongst others) . However, experimental psychological research is more than survey data. Especially in cognitive psychology, the study of mental processes in human beings, one of the most common approaches is to measure reactions to stimuli that are usually presented on a computer screen, thereby gaining an external measurement of internal processes. The two main variables of such psychophysical measurements are error rates (ERs) and reaction times (RTs). In a typical cognitive experiment, a participant is presented with a display of different conditions and reacts to this display via a keypress. Usually, this is done by inviting participants into the lab, seating them in a testing chamber, and instructing them to press a mouse or keyboard button when appropriate. Through this approach researchers can control for external factors like hardware (computer system, keyboard, and mouse), software (operating system, programming language, versions), and the environment (e.g., noise, lightning, distractions). On the other hand, conducting cognitive experiments in a classical in-lab setting requires a high amount of resources, i (...truncated)


This is a preview of a remote PDF: https://link.springer.com/content/pdf/10.3758%2Fs13428-016-0783-4.pdf

Kilian Semmelmann, Sarah Weigelt. Online psychophysics: reaction time effects in cognitive experiments, Behavior Research Methods, 2017, pp. 1241-1260, Volume 49, Issue 4, DOI: 10.3758/s13428-016-0783-4