Researchers who employ online data collection from human subjects currently face a conundrum: It is both essential to how behavioral science functions and threatened by low-quality data. It is often assumed that random, inconsistent, and otherwise incomprehensible data in online surveys comes mainly from bots. Despite this assumption, few studies have directly examined where problematic data comes from, even though identifying the source has important implications for creating the right solutions. We examined this issue on several popular participant-recruitment platforms, including Mechanical Turk (MTurk) and Lucid. Across four studies spanning 5 years using multiple methods, we here provide evidence that most of the data-quality problems affecting online research using online panels can be tied to fraudulent users from outside of the United States-not bots. We identify many of the telltale signs that humans leave behind and describe the most effective ways of blocking problematic human responses to address the online data-quality problem.

[This corrects the article DOI: 10.1371/journal.pone.0226394.].

Predictions concerning upcoming visual input play a key role in resolving percepts. Sometimes input is surprising, under which circumstances the brain must calibrate erroneous predictions so that perception is veridical. Despite the extensive literature investigating the nature of prediction error signalling, it is still unclear how this process interacts with the functionally segregated nature of the visual cortex, particularly within the temporal domain. Here, we recorded electroencephalography (EEG) from humans (N = 32) whilst they viewed static image trajectories containing a bound object that sequentially changed along different visual attribute dimensions (shape and colour). Crucially, the context of this change was designed to appear random (and unsurprising) or violate the established trajectory (and cause a surprise). Event-related potential analysis found no effects of surprise after controlling for cortical adaptation. However, multivariate pattern analyses found whole-scalp neural representations of visual surprise that overlapped between attributes, albeit at distinct, attribute-specific latencies. These findings suggest that visual surprise results in generalised (i.e., attribute-agnostic) prediction error responses that conform to an attribute-dependent temporal hierarchy.

Onset primacy is a behavioural phenomenon whereby humans identify the appearance of an object (onset) with greater efficiency than other kinds of visual change, such as the disappearance of an object (offset). The default mode hypothesis explains this phenomenon by postulating that the attentional system is optimised for onset detection in its initial state. The present study extended this hypothesis by combining a change-detection task and measurement of the P300 event-related potential, which was thought to index the amount of processing resources available to detecting onsets and offsets. In an experiment, while brain activity was monitored by electroencephalography, participants indicated the locations of onsets and offsets under the condition in which they occurred equally often in the same locations across trials. Although there was no reason to prioritise detecting one type of change over the other, onsets were detected more quickly, and they evoked a larger P300 than offsets. These results suggest that processing resources are preferentially allocated to onset detection. This biased allocation may be a basis on which the attentional system defaults to the 'onset detection' mode.

The foetal period constitutes a critical stage in the construction and organisation of the mammalian nervous system. In recent work, we have proposed that foetal brain development is structured by bottom-up (interoceptive) inputs from spontaneous physiological rhythms such as the heartbeat (Corcoran et al., 2023). Here, we expand this visceral afferent training hypothesis to incorporate the development of top-down (allostatic) control over bodily states. We conceptualise the emergence of cardiac regulation as an early instance of sensorimotor contingency learning that scaffolds the development of agentic control. We further propose that the brain’s capacity to actively modify and regulate the afferent feedback it receives through interoceptive (and other) channels – and to parse these signals into their self-generated (reafferent) and externally-generated (exafferent) components – is crucial for grounding the distinction between self and other. Finally, we explore how individual differences in the ways these training regimes are implemented (or disrupted) might impact developmental trajectories in gestation and infancy, potentiating neurobehavioural diversity and disease risk in later life.

The multifaceted nature of subjective experience poses a challenge to the study of consciousness. Traditional neuroscientific approaches often concentrate on isolated facets, such as perceptual awareness or the global state of consciousness and construct a theory around the relevant empirical paradigms and findings. Theories of consciousness are, therefore, often difficult to compare; indeed, there might be little overlap in the phenomena such theories aim to explain. Here, we take a different approach: starting with active inference, a first principles framework for modelling behaviour as (approximate) Bayesian inference, and building up to a minimal theory of consciousness, which emerges from the shared features of computational models derived under active inference. We review a body of work applying active inference models to the study of consciousness and argue that there is implicit in all these models a small set of theoretical commitments that point to a minimal (and testable) theory of consciousness.

Polar angle asymmetries (PAAs), the differences in perceptual experiences and performance across different regions of the visual field are present in various paradigms and tasks of visual perception. Currently, research in this area is sparse, particularly regarding the influence of PAAs during perceptual illusions, highlighting a gap in visual cognition studies. We aim to fill this gap by measuring PAAs across the visual field during an illusion applied to test conscious vision widely. Motion-induced blindness (MIB) is an illusion when a peripheral target disappears from consciousness as the result of a continuously moving background pattern. During MIB we separately measured the average disappearance time of peripheral targets in eight equidistant visual field positions. Our results indicate a significant variation in MIB disappearance times and frequencies as a function of target location. Specifically, we found shorter and fewer disappearances along the cardinal compared to oblique directions, and along the horizontal compared to the vertical meridian. Our results suggest specific consistencies between visual field asymmetries and conscious visual perception.

Behavioral scientists looking to run online studies are confronted with a bevy of options. Where to recruit participants? Which tools to use for survey creation and study management? How to maintain data quality? In this tutorial, we highlight the unique capabilities of market-research panels and demonstrate how researchers can effectively sample from such panels. Unlike the microtask platforms most academics are familiar with (e.g., MTurk and Prolific), market-research panels have access to more than 100 million potential participants worldwide, provide more representative samples, and excel at demographic targeting. However, efficiently gathering data from online panels requires integration between the panel and a researcher’s survey in ways that are uncommon on microtask sites. For example, panels allow researchers to target participants according to preprofiled demographics (“Level 1” targeting, e.g., parents) and demographics that are not preprofiled but are screened for within the survey (“Level 2” targeting, e.g., parents of autistic children). In this article, we demonstrate how to sample hard-to-reach groups using market-research panels. We also describe several best practices for conducting research using online panels, including setting in-survey quotas to control sample composition and managing data quality. Our aim is to provide researchers with enough information to determine whether market-research panels are right for their research and to outline the necessary considerations for using such panels.

Predictive coding theories assert that perceptual inference is a hierarchical process of belief updating, wherein the onset of unexpected sensory data causes so-called prediction error responses that calibrate erroneous inferences. Given the functionally specialised organisation of visual cortex, it is assumed that prediction error propagation interacts with the specific visual attribute violating an expectation. We sought to test this within the temporal domain by applying time-resolved decoding methods to electroencephalography (EEG) data evoked by contextual trajectory violations of either brightness, size, or orientation within a bound stimulus. We found that following ∼170 ms post stimulus onset, responses to both size violations and orientation violations were decodable from physically identical control trials in which no attributes were violated. These two violation types were then directly compared, with attribute-specific signalling being decoded from 265 ms. Temporal generalisation suggested that this dissociation was driven by latency shifts in shared expectation signalling between the two conditions. Using a novel temporal bias method, we then found that this shared signalling occurred earlier for size violations than orientation violations. To our knowledge, we are among the first to decode expectation violations in humans using EEG and have demonstrated a temporal dissociation in attribute-specific expectancy violations.

To understand human behavior, social scientists need people and data. In the last decade, Amazon's Mechanical Turk (MTurk) emerged as a flexible, affordable, and reliable source of human participants and was widely adopted by academics. Yet despite MTurk's utility, some have questioned whether researchers should continue using the platform on ethical grounds. The brunt of their concern is that people on MTurk are financially insecure, subject to abuse, and earn inhumane wages. We investigated these issues with two representative probability surveys of the U.S. MTurk population (N = 4094). The surveys revealed: (1) the financial situation of people on MTurk mirrors the general population, (2) most participants do not find MTurk stressful or requesters abusive, and (3) MTurk offers flexibility and benefits that most people value above other options for work. People reported it is possible to earn more than $10 per hour and said they would not trade the flexibility of MTurk for less than $25 per hour. Altogether, our data are important for assessing whether MTurk is an ethical place for research.

Embodied cognition-the idea that mental states and processes should be understood in relation to one's bodily constitution and interactions with the world-remains a controversial topic within cognitive science. Recently, however, increasing interest in predictive processing theories among proponents and critics of embodiment alike has raised hopes of a reconciliation. This article sets out to appraise the unificatory potential of predictive processing, focusing in particular on embodied formulations of active inference. Our analysis suggests that most active-inference accounts invoke weak, potentially trivial conceptions of embodiment; those making stronger claims do so independently of the theoretical commitments of the active-inference framework. We argue that a more compelling version of embodied active inference can be motivated by adopting a diachronic perspective on the way rhythmic physiological activity shapes neural development in utero. According to this visceral afferent training hypothesis, early-emerging physiological processes are essential not only for supporting the biophysical development of neural structures but also for configuring the cognitive architecture those structures entail. Focusing in particular on the cardiovascular system, we propose three candidate mechanisms through which visceral afferent training might operate: (a) activity-dependent neuronal development, (b) periodic signal modeling, and (c) oscillatory network coordination.

Survey respondents who are non-attentive, respond randomly, or misrepresent who they are can impact the outcomes of surveys. Prior findings reported by the CDC have suggested that people engaged in highly dangerous cleaning practices during the COVID-19 pandemic, including ingesting household cleaners such as bleach. In our attempts to replicate the CDC's results, we found that 100% of reported ingestion of household cleaners are made by problematic respondents. Once inattentive, acquiescent, and careless respondents are removed from the sample, we find no evidence that people ingested cleaning products to prevent a COVID-19 infection. These findings have important implications for public health and medical survey research, as well as for best practices for avoiding problematic respondents in all survey research conducted online.

This paper introduces Connect, CloudResearch's innovative platform designed to revolutionize the realm of online participant recruitment in social and behavioral science research. Operating as a marketplace, Connect facilitates interactions between researchers and participants, enabling the deployment of surveys and experiments constructed via third-party tools such as Qualtrics, SurveyMonkey, and Google forms. With its current focus on the U.S. demographic (with plans of future expansion to other English-speaking countries) and those aged 18 and above, Connect proves versatile in accommodating a diverse range of studies, including academic, market, and user experience research, as well as machine learning. Connect's uniqueness lies in its tri-fold emphasis on advanced features, data quality, and affordability. Advanced attributes include collaborative tools, a flexible API, and capabilities supporting intricate study designs. To assure impeccable data quality, Connect incorporates Sentry®, CloudResearch’s proprietary participant vetting mechanism, coupled with stringent technical evaluations. Despite these advancements, Connect remains cost-effective, charging researchers the lowest service fee in the online recruitment sector. This paper delves deeper into the platform’s attributes, with particular attention to its emphasis on participant experience, advanced functionalities, and the robust data quality assurance methods in place.

People in online studies sometimes misrepresent themselves. Regardless of their motive for doing so, participant misrepresentation threatens the validity of research. Here, we propose and evaluate a way to verify the age of online respondents: a test of era-based knowledge. Across six studies (N = 1543), participants of various ages completed an age verification instrument. The instrument assessed familiarity with cultural phenomena (e.g., songs and TV shows) from decades past and present. We consistently found that our instrument discriminated between people of different ages. In Studies 1a and 1b, self-reported age correlated strongly with performance on the instrument (mean r = .8). In Study 2, the instrument reliably detected imposters who we knew were misrepresenting their age. For impostors, self-reported age did not correlate with performance on the instrument (r = .077). Finally, in Studies 3a, 3b, and 3c, the instrument remained robust with African Americans, people from low educational backgrounds, and recent immigrants to the United States. Thus, our instrument shows promise for verifying the age of online respondents, and, as we discuss, our approach of assessing "insider knowledge" holds great promise for verifying other identities within online studies.

Sometimes agents choose to occupy environments that are neither traditionally rewarding nor worth exploring, but which rather promise to help minimise uncertainty related to what they can control. Selecting environments that afford inferences about agency seems a foundational aspect of environment selection dynamics - if an agent can't form reliable beliefs about what they can and can't control, then they can't act efficiently to achieve rewards. This relatively neglected aspect of environment selection is important to study so that we can better understand why agents occupy certain environments over others - something that may also be relevant for mental and developmental conditions, such as autism. This online experiment investigates the impact of uncertainty about agency on the way participants choose to freely move between two environments, one that has greater irreducible variability and one that is more complex to model. We hypothesise that increasingly erroneous predictions about the expected outcome of agency-exploring actions can be a driver of switching environments, and we explore which type of environment agents prefer. Results show that participants actively switch between the two environments following increases in prediction error, and that the tolerance for prediction error before switching is modulated by individuals' autism traits. Further, we find that participants more frequently occupy the variable environment, which is predicted by greater accuracy and higher confidence than the complex environment. This is the first online study to investigate relatively unconstrained ongoing foraging dynamics in support of judgements of agency, and in doing so represents a significant methodological advance.

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. In a pre-registered study, participants ( N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk’s Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.

Identifying the faces of familiar persons requires the ability to assign several different images of a face to a common identity. Previous research showed that the occipito-temporal cortex, including the fusiform and the occipital face areas, is sensitive to personal identity. Still, the viewpoint, facial expression and image-independence of this information are currently under heavy debate. Here we adapted a rapid serial visual stimulation paradigm Johnston et al. (2016, https://doi.org/10.1016/j.cortex.2016.10.002) and presented highly variable ambient-face images of famous persons to measure functional magnetic resonance imaging (fMRI) adaptation. fMRI adaptation is considered as the neuroimaging manifestation of repetition suppression, a neural phenomenon currently explained as a correlate of reduced predictive error responses for expected stimuli. We revisited the question of image-invariant identity-specific encoding mechanisms of the occipito-temporal cortex, using fMRI adaptation with a particular interest in predictive mechanisms. Participants were presented with trials containing eight different images of a famous person, images of eight different famous persons or seven different images of a particular famous person followed by an identity change to violate potential expectation effects about person identity. We found an image-independent adaptation effect of identity for famous faces in the fusiform face area. However, in contrast to previous electrophysiological studies, using similar paradigms, no release of the adaptation effect was observed when identity-specific expectations were violated. Our results support recent multivariate pattern analysis studies, showing image-independent identity encoding in the core face-processing areas of the occipito-temporal cortex. These results are discussed in the frame of recent identity-processing models and predictive mechanisms.

Online data collection has become indispensable to the social sciences, polling, marketing, and corporate research. However, in recent years, online data collection has been inundated with low quality data. Low quality data threatens the validity of online research and, at times, invalidates entire studies. It is often assumed that random, inconsistent, and fraudulent data in online surveys comes from ‘bots.’ But little is known about whether bad data is caused by bots or ill-intentioned or inattentive humans. We examined this issue on Mechanical Turk (MTurk), a popular online data collection platform. In the summer of 2018, researchers noticed a sharp increase in the number of data quality problems on MTurk, problems that were commonly attributed to bots. Despite this assumption, few studies have directly examined whether problematic data on MTurk are from bots or inattentive humans, even though identifying the source of bad data has important implications for creating the right solutions. Using CloudResearch’s data quality tools to identify problematic participants in 2018 and 2020, we provide evidence that much of the data quality problems on MTurk can be tied to fraudulent users from outside of the U.S. who pose as American workers. Hence, our evidence strongly suggests that the source of low quality data is real humans, not bots. We additionally present evidence that these fraudulent users are behind data quality problems on other platforms.

Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Copy DOI