Existing research on group-based appeals primarily uses text-based methods, and while many studies show the importance of visuals in implicitly cueing groups, this data is rarely captured in a systematic way. This paper seeks to make the first important step towards filling this gap by outlining a coding scheme to evaluate how group-based appeals are used multimodally in modern political campaigns. This paper builds categories from a qualitative sample of 182 images taken from 28 television and 63 Facebook ads from candidates running in the US 2020 House of Representatives elections. Direct appeals are captured as explicit group mentions and I present new categories for indirect and baseline appeals, which incorporate primarily visual indicators of groups. Intercoder reliability tests were conducted, and the schema was applied to a larger sample of 2480 images from 125 television ads from candidates running in the three most populous states (California, Texas, Florida). This paper finds that candidates use direct and indirect appeals at similar rates, often using them in combination. Capturing visual data therefore enables greater coverage of the range of group-based appeals that political campaigns conduct. Secondly, candidates are more likely to cue occupational groups indirectly, and capturing only direct cues may lead to skewed findings in terms of which groups candidates appeal to. I find that this new coding scheme may reduce bias in measures of both the prevalence of group-based appeals and the types of groups that campaigns appeal to in modern political discourse.
If survey respondents do not interpret a question as it was intended, they may, in effect, answer the wrong question, increasing the chances of inaccurate data. Researchers can bring respondents' interpretations into alignment with what is intended by defining the terms that respondents are at risk of misunderstanding. This article explores strategies to increase alignment between researchers' intentions and respondents' answers by taking advantage of the unique affordances of online surveys compared to paper or other analog formats. Web surveys are often text-based, but allow for the seamless integration of embedded audio material so that users may read, listen to, or both read and listen to survey instructions. Unimodal definitions are either spoken or textual, while multimodal definitions are both spoken and textual. Further, definitions can be designed to take advantage of the affordances of each mode. While mode-invariant definitions contain the same words irrespective of whether they are textual or spoken, mode-optimized definitions are designed to take advantage of the affordances of written and spoken communication. For example, definitions optimized for textual presentation use fewer words than corresponding mode-invariant definitions and are designed so the key information is visually salient, while definitions optimized for spoken presentation are shorter and more colloquial than corresponding mode-invariant definitions. In this study, both mode-optimized and mode-invariant formats improved alignment. Multimodal, mode-optimized definitions produced improved alignment over both types of unimodal definitions. This study suggests that multimodal definitions, when thoughtfully designed, can improve data quality in online surveys without negatively impacting respondents.
Understanding how people seek and apply for jobs online is crucial for addressing social inequality, discrimination, and aiding companies in attracting suitable candidates. Conventional surveys struggle to capture the nuances of online job searches that, as many online events, are characterized by repetition, low distinctiveness, and limited emotional impact. These characteristics can lead to memory-related errors, becoming more likely as the time between the event and the survey increases. Passively collected data, such as metered data provided by online panel members who install tracking software on their browsing devices, offer an alternative. While these data provide objective insights into online job searches, they suffer other types of errors, and cannot capture subjective information and all potential objective data of interest. This paper explores an alternative approach: sending surveys to individuals in a metered panel shortly after an event of interest is detected through metered data. These "in-the-moment" surveys aim to fill in missing information not obtainable through passive data collection while reducing memory-related errors that affect conventional surveys. To assess the feasibility and benefits of this method, an experiment comparing in-the-moment surveys triggered by online job applications with conventional surveys was conducted in an opt-in online panel in Spain to research how people apply for a job online. The results reveal that metered panelists accept well in-the-moment surveys, displaying high participation levels and positive evaluations regarding effort and satisfaction, without perceiving an increased privacy risk. Moreover, the data indicate positive impacts on data quality, with longer and more detailed responses to open-ended questions. However, not all aspects saw substantial improvements, with the reduction of non-recall being weaker than expected, possibly due to participants' overconfidence in their memories. The significant disparities observed in substantive results between both types of surveys also suggest that participants are not fully aware of what they do not remember.
The studies reported here explore how the "self-view" window (a live video feed of oneself) affects live video survey respondents' likelihood of disclosing sensitive information and their feelings about the interview. In Study 1 (2012), 124 laboratory respondents answered sensitive and nonsensitive questions taken from US government and social scientific surveys over Skype, either with or without a self-view window. Respondents randomly assigned to having a self-view disclosed no less sensitive information than those without a self-view, and on a few questions, they disclosed more (more frequent alcohol use and more sex partners). Self-view respondents also perceived the interview as less sensitive, and they reported less copresence with the interviewer, reduced selfconsciousness, and greater comfort answering many of the sensitive questions. Study 2 (2017) replicates these findings in a second sample of 133 respondents by (a) tracking where video survey respondents look on the screen-at the interviewer, at the self-view, or elsewhere-while answering the same survey questions and (b) examining how gaze location and duration differ for sensitive vs. nonsensitive questions and for more and less socially desirable answers. Findings include that self-view respondents looked less at the self-view while answering sensitive (vs. nonsensitive) questions, and that respondents who looked more at the self-view window reported feeling less self-conscious and less worried about how they presented to the interviewer. Results demonstrate that the self-view can change respondents' experience and where they look during a video interview. They also document, for the first time in video surveys, surprising individual variability in looking at the self-view, with some respondents never once looking and others looking at their self-view as much as 50% of the time. Attending to how self-view and respondents' choices (e.g., turning it on or off) affect respondent experience and data quality will be important as live video surveys are increasingly deployed.
The integration of video interviewing in survey research is relatively new and may offer similar benefits as telehealth visits in mental health research. Methodological evaluations of video interviewing are needed for large-scale surveys. Over 3,000 clinical interviews were conducted by video and over 1,500 by phone for a national study of U.S. adults, the Mental and Substance Use Disorders Prevalence Study (MDPS). Sociodemographic differences were observed among those who completed a clinical interview by video compared to phone respondents. Higher prevalence rates of all disorders, with the exception of schizophrenia spectrum disorders, were found for video respondents. Higher prevalence rates of generalized anxiety disorder (video 11.3% vs. 8.0%, p < .05), bipolar 1 (2.1% vs. 0.7%, p < .05) and obsessive compulsive disorder (OCD; 3.1% vs. 1.5%, p < .05) were observed among those completing an interview by video compared to those interviewed by phone. Individual logistic regression models were calculated for each disorder adjusting for sociodemographic characteristics to assess the difference in prevalence rate by mode of interview. Respondents interviewed by video had higher odds of having bipolar 1 (OR = 2.96, 95% CI [1.42, 6.17]), OCD (OR = 2.16, 95% CI [1.20, 3.90])and having two or more mental health disorders (OR = 1.64, 95% CI [1.23, 2.19]) than those interviewed by phone after adjusting for sociodemographic characteristics. While further investigation using experimental approaches is required, video interviewing may improve the ability to detect mental health conditions in large-scale survey research.
Researchers are increasingly using social media platforms for survey recruitment. However, empirical evidence remains sparse on how the content and design characteristics of advertisements used for recruitment affect response quality in surveys. Building on leverage-salience and self-determination theory, we assess the effects of advertisement design on response quality. We argue that different advertisement designs may resonate with specific social groups who vary in their commitment to the survey, resulting in differences in the observed response quality. We use data from a study conducted via ads placed on Facebook in Germany and the United States in June 2023. The survey, focusing on attitudes toward climate change and immigration, featured images with varying thematic associations with the topics (strong, loose, neutral). The sample consisted of 4,170 respondents in Germany and 5,469 respondents in the United States. We compare several data quality indicators, including break-off rate, completion time, non-differentiation, item non-response, passing an attention check question, and follow-up availability, across different advertisement features. Regression analyses indicate differences in response quality across advertisement designs, with a strong thematic design generally being associated with poorer response quality. Strongly themed ad designs are generally associated with higher attrition, non-differentiation, and item non-response, and with a lower probability of passing an attention check and providing an e-mail address for future survey inquiries. Our study advances the literature by highlighting the substantial impact of advertisement design on survey data quality, and emphasizing the importance of tailored decision-making in recruitment design for social media-based survey research.
Modern predictive modeling tools, such as random forests (and related ensemble methods), have become almost ubiquitous in research applications involving innovative combinations of survey methodology and data science. However, an important potential flaw in the widespread application of these methods has not received sufficient research attention to date. Researchers at the junction of computer and survey science frequently leverage linked data sets to study relationships between variables, where the techniques used to link two (or more) data sets may be probabilistic and non-deterministic in nature. If frequent mismatch errors occur when linking two (or more) data sets, the commonly desired outputs of predictive modeling tools describing relationships between variables in the linked data sets (e.g., variable importance, confusion matrices, RMSE, etc.) may be negatively affected, and the true predictive performance of these tools may not be realized. We demonstrate a new methodology based on mixture modeling that is designed to adjust modern predictive modeling tools for the presence of mismatch errors in a linked data set. We evaluate the performance of this new methodology in an application involving the use of observed Twitter/X activity measures and predicted socio-demographic features of Twitter/X users to accurately predict linked measures of political ideology that were collected in a designed survey, where respondents were asked for consent to link any Twitter/X activity data to their survey responses (exactly, based on Twitter/X handles). We find that the new methodology, which we have implemented in R, is able to largely recover results that would have been seen prior to the introduction of mismatch errors in the linked data set.
Today, an increasing number of surveys offer respondents the choice of which language they want to answer the questionnaire. In later data analysis, however, the language in which the respondent answers the questions is often ignored, and no distinction is made regarding whether that language is the respondent's mother tongue. Several psychological theoretical considerations and empirical observations indicate that respondents' answering behaviors are influenced by whether the questions are presented in their mother tongue or a non-native language. Therefore, the extent to which these mechanisms and effects of language used are also applicable and relevant in social science studies remains unclear. Based on models of cognitive load, satisficing, and language-dependent memory, the influence of language nativeness on response behavior is explained from a theoretical point of view. The research question will be answered by analyzing the data from the refugee study ReGES (Refugees in the German Educational System). The results of the analyses show that there is a difference in response behavior depending on whether a question is answered in a mother tongue or a non-native language. The implications, both from a survey methodological point of view and for further research, will be discussed.
Cognitive interviewing is widely used to pretest survey questionnaires and is considered a best practice (e.g., Willis, 2005, 2018; Beatty & Willis, 2007). However, the method has been controversial because, among other concerns, it requires interviewers to probe respondents for more detail or clarity about their experience answering draft survey questions which may lead them to report "problems" they have not actually experienced (e.g., Conrad & Blair, 2009). The present study investigates this possibility from the perspective of Acquiescent Response Style (ARS) - the tendency for survey respondents to select positive responses such as "yes" or "strongly agree," irrespective of the question's content (e.g., Baumgartner & Steenkamp, 2001). For example, respondents in a cognitive interview might affirm experiencing a problem mentioned in or implied by an interviewer's probe even if they have not actually experienced it. We embedded a probing experiment in a cognitive interview pretest of a health survey in which respondents participated in cognitive interviews that used either directive probes (n=41) or non-directive probes (n=26). Directive probes explicitly queried respondents about a specific, intentionally unlikely interpretation of each question in a draft questionnaire; non-directive probes were open-ended. Directive probe (DP) respondents affirmed the interpretation queried in the probes over five times more often than respondents in the non-directive probe (NP) group volunteered these interpretations. This pattern was reversed for interpretations of the questions that were volunteered, i.e., about which DP respondents were not asked: NP respondents volunteered alternative interpretations over four times more than DP respondents. These effects were particularly pronounced for respondents with lower levels of education and who were younger. The findings suggest that directive probing in cognitive interviewing can promote responding that is reminiscent of ARS - an affirmation bias - and likely harmful for the quality of evidence produced in cognitive interviews.
Panel surveys suffer from attrition, where participants drop out over time. To maintain generalizability, refreshment samples are frequently employed, bringing in new individuals, increasing the number of panelists, and balancing sample composition. Although refreshment samples offer numerous advantages, the inclusion of new panel members may introduce bias into the analysis if the design weights are not appropriately tailored to these new members and adjusted to align with existing panel members. If not correctly accounted for, their inclusion may bias results. This paper addresses the issue of designing proper weights by applying the multiple-frame weighting approach proposed by Kalton and Anderson, which is generally used for cross-sectional surveys, to ongoing panel studies with refreshment samples. We demonstrate its application to a synthetic data set and a probability-based mixed-mode panel with an initial sample and two refreshment samples. We compare estimates obtained using multiple-frame weighting with those obtained using unweighted and naively weighted methods (where design weights are used as calculated for the respective samples without adjusting for the fact that some members of the population have a chance of being sampled more than once due to the refreshments). These comparisons showcase the potential for bias introduced by neglecting proper weighting and underscore the importance of both a multiple-frame weighting approach and meticulous sample documentation.
The collection of photos through online surveys has emerged as a valuable research tool given the growing use of smartphones, which have facilitated the capture and share of photos. However, gaps persist in understanding respondents' involvement in these tasks when asked to perform them in an online survey. Existing literature lacks insights into participants' preferences, their assessment of questions asking for photos, and how their characteristics might impact their participation in such queries. This paper addresses these gaps, while also comparing how image-based formats compare to conventional ones. Conducted among 1,270 parents living with children in primary school of an opt-in panel in Spain, the mobile online survey implemented in this study revealed a preference for conventional questions, and higher participation in that format than in the image-based one. Respondents able to choose their response format and preferring images presented higher participation rates than those without a choice. While both formats were perceived as equally easy, participants using conventional formats liked the questions better than those answering through photos. Finally, age, being female, having a tertiary education degree, and using the camera at least once a week positively impacted the participation in image-based questions, whereas comfort with new technologies increased the likelihood of liking this format. This study not only fills critical gaps in the literature but also sheds light on the complexities of asking for photos in online surveys.
This research explores the potential of augmented Data Download Packages (aDDPs) as a novel approach to analyze digital trace data, using TikTok as a use case to demonstrate the broader applicability of the method. The study demonstrates how these data packages can be used in social science research to understand better user behavior, content consumption patterns, and the relationship between self-reported preferences and actual digital behavior. We introduce the concept of aDDPs, which extend the conventional Data Download Packages (DDPs) by augmenting the collected data with survey data, metadata, content data, and multimodal content embeddings, among other possibilities- rendering aDDPs an unprecedentedly rich data source for social science research. This work provides an overview and guidance on collecting, augmenting DDPs, and analyzing the resulting aDDPs. In a pilot study on 18 aDDPs, we use the combination of data components in aDDPs to facilitate research on user engagement behavior and content classification. We showcase the potential of the information breadth and depth that aDDPs depict by exploiting the combination of multimodal content embeddings, the users' watch history, and survey data. To do so, we train and compare uni- and multimodal classifiers, classify the 18 aDDPs' videos, and investigate the extent to which user engagement behavior impacts future content suggestions. Furthermore, we compare the users retrieved content with the users' self-reported content consumption.
This Research Note reports on a list experiment regarding anti-immigrant sentiment (n=1,965) that was fielded in Spain in 2020. Among participants with left-of-center ideol-ogy, the experiment originated a negative difference-in-means between treatment and control. Drawing on Zigerell's (2011) def lation hypothesis, we assess the possibility that leftist treatment group respondents may have altered their scores by more than one to distance themselves unmistakably from the sensitive item. We consider this possibility plausible in a context of intense polarization where immigration attitudes are closely associated with political ideology. This study's data speak to the results of recent meta-analyses that have revealed list-experiments to fail when applied to prejudiced attitudes and other highly sensitive issues - i.e., precisely the kind of issues with regard to which the technique ought to work best. We conclude that the possibility of strategic response error in specific respondent categories needs to be considered when staging and inter-preting list experiments
The number of studies assessing measurement invariance of the European Social Sur-vey's (ESS) immigration scale increased in recent years. However, the comparability of findings is limited due to the lack of consistency in the analytic strategies and methods employed across these studies. The present study aims to address this issue by employ-ing a consistent approach: a multigroup confirmatory factor analysis (MGCFA), to test for measurement invariance of attitudes towards immigration in each of the first nine rounds of the ESS. Moreover, we estimate the measurement quality by computing the reliability coefficient Omega in each country in each round of the ESS.Our results reveal that metric invariance holds for all countries but one (Finland) in all rounds, indicating that covariances and regression coefficients can be compared meaningfully. While scalar invariance only holds for different subgroups of countries within each round, partial invariance is fulfilled in all countries, meaning that at least one indicator is equal for all countries allowing for latent mean comparisons. Further-more, assessing the measurement quality, we find the attitudes towards immigration index similarly good across the different countries and rounds.
Integrating voice inputs into web surveys holds the potential for various benefits, in-cluding eliciting more comprehensive and elaborate responses or extracting additional information from vocal tones and ambient sounds. Nevertheless, important challenges persist, including technical problems, privacy concerns, and low participation rates. Given the limited knowledge on this subject, this research note addresses four research questions, distinguishing between two voice input methods (dictation and voice re-cording) and two approaches to presenting them (providing a choice, or pushing re-spondents toward voice inputs, with a text alternative offered only in the absence of response): RQ1. What reasons are provided for not opting for voice inputs when they are offered? RQ2. Which variables are associated with the reported use of voice inputs? RQ3.What challenges do individuals answering through voice inputs report? And RQ4. How do respondents evaluate the different methods of answering they employed?Drawing on data from a survey on nursing homes conducted in February/March 2023 within the Netquest opt-in online panel in Spain (1,001 completes), where participants were offered to respond to two experimental questions through voice methods, our analyses reveal that contextual factors and the perceived challenge of oral expression are key reasons for abstaining from voice input responses. Furthermore, individuals who exhibited complete trust in the confidentiality of their responses and those already using voice input in their daily lives were significantly more likely to opt for voice in-puts. Among respondents utilizing voice inputs, recurring challenges included contex-tual constraints and difficulties in verbal expression, alongside technical problems. Despite these hurdles, a majority of participants found answering through voice easy, although a lower proportion reported liking it. These results contribute to the limited literature and can help enhance the effectiveness of voice input surveys.
Background : Criminological research shows that there is nearly always a strong and positive association between delinquency and being a victim of crime. This so-called victim-offender overlap is one of the most consistent and best documented findings in criminology. However, examinations using longitudinal panel data are rather scarce. Previous analyses based on latent growth and cross-lagged panel models showed that the developments of victimization and offending are parallel processes that expose similar stability and mutual influence over the period of adolescence and early adulthood (Erdmann & Reinecke, 2018). Objectives : The present study examines the relationship between victimization and offending over the phase of adolescence and emerging adulthood. The focus is on the application of continuous time dynamic modeling and on comparing results using data from the criminological panel study Crime in the Modern City . For the present analyses, seven consecutive panel waves are used that contain information about German adolescents from the age of 14 to 20 years. Approach : The relationship between victimization and offending is analyzed by con - tinuous time structural equation modeling using the R package ctsem (Driver & Voelkle, 2018, 2021). In addition to the unconditional models, relevant predictors (gender, routine activities) are considered in the conditional models. Methododological and substantive as - pects of continuous time dynamic modeling are highlighted in the discussion of the results.
The assignment of questionnaires between the 13 survey waves in the panel study "Crime in the Modern City" (CrimoC) was done by matching self-generated codes. This method was challenging because the individual codes tend to be ambiguous, prone to errors and the resulting panel data can be biased. The individual data were merged over time using an error-tolerant matching process with manual handwriting comparison. Despite these problems, there is no alternative to the chosen method with regard to anonymity and data protection. Until now, the self-generated codes of each new survey wave were matched against the codes of the last and second-last wave. Over the years, this led to an increasing discrepancy between the data originally collected and the data linked to the panel. For this reason, first in a pretest and later for the complete sample, the cases that had not yet been linked to the panel were subsequently matched with earlier waves. This panel consolidation proved to be very successful. A total of 3,589 original missing units were subsequently filled with case data. This paper describes the steps taken to optimize the quality of the panel data set and illustrates exemplarily on specified criteria which properties of the panel data set could be improved. Since the importance of panel studies is steadily increasing in social science research this paper is relevant for researchers who need to make matching decisions within panel studies. Assurance of anonymity can counteract panel attrition. Self-generated codes represent one possibility in this regard, and are discussed in terms of feasibility and effectiveness.
Recent studies use Fixed Effects (FE) models to estimate the causal effect of obesity on socioeconomic status, the so-called obesity penalty. In this paper, I will illustrate the advantages of using a Difference in Differences (DID) approach as an alternative method of causal analysis. Combining the German National Health Interview and Examination Survey 1998 (GNHIES98) and the German Health Interview and Examination Survey for Adults 2008 (DEGS1) allowed for a panel analysis of 3934 respondents. The dependent variable is a socioeconomic status score that integrates level of education, occupation and household income. The binary treatment variable is abdominal obesity. To estimate the causal effect of the treatment, FE and DID approaches were used.Both the FE model and the DID estimate show no statistically significant causal effect of abdominal obesity on socioeconomic status for adults in Germany. However, both the respondents who became obese and those who stayed non-obese experience a rise in socioeconomic status over time. Nonetheless, the non-obese group had a more substantial increase in socioeconomic status than the obese group. Therefore, the obesity penalty does not necessarily have to be a decrease in socioeconomic status but could instead be a slowed growth or stagnation in status. The advantage of the DID approach is that the development in the control group is explicit. If obese individuals are more likely to have less favorable positive trends in socioeconomic status over time than other individuals, using DID estimates demonstrates the obesity penalty more effectively than using only FE models.
Background: The aim of the study is to investigate the longitudinal and cross-cultural measurement invariance of the Short-Form 12-Item Health Survey (SF-12) between Native Germans, European migrants and Non-European Migrants. Further, we test for differences in latent means dependent on invariance restrictions. Methods: We include 7 waves (2006-2018) from a representative panel study in Germany. We apply Multigroup Confirmatory Factor Analysis via a Structural Equation Modelling approach. Finally, we compare gender and age adjusted latent means between different settings of invariance assumptions. Results: The decrease in model fit measures by increasing equality constraints on the SF-12 factor structure of both physical and mental health between origin groups and across time is within common thresholds for good model fit. Latent means of both health factors differ, dependent on whether scalar invariance is set longitudinally and cross-culturally, or only longitudinally. Conclusion: We conclude acceptable longitudinal and cross-cultural measurement invariance of the SF-12 for a period of 12 years. Yet, ignoring multigroup scalar invariance constraints produces bias in the latent means of both health factors, where migrant health is shown to be overestimated, especially for Non-European migrants if indicator intercepts are not sufficiently constrained.
