Online platforms prioritize long-term business outcomes, yet typical experiments are far too short to measure these outcomes directly. Our goal in this paper is to collect and share industry knowledge on how to make decisions from short-term experiments that are better aligned with long-term outcomes. Based on a daylong workshop with 26 experts from 15 online platforms and 4 universities, we formulate a series of propositions that reflect current industry knowledge. Participants largely agreed that reversals of sign from short-run to long-run treatment effects are rare, with reversals concentrating in specific cases such as treatments involving content quality signals, hyper-monetization, and pricing. Although the magnitude of treatment effects can shift over time, a "univariate autosurrogate", corresponding to the short-run treatment effect on the long-run metric of interest, is often hard to beat. A recurring theme was the importance of surrogates that are not only (or even primarily) unbiased for true long-run outcomes, but that improve decision-making. Thus, participants generally agreed that simple, interpretable surrogates were generally preferable to elaborate but hard-to-explain surrogate indices. Participants also agreed that, due to concerns about confounding and transportability, experimentally-learned surrogates are generally preferable to observationally-learned surrogates. However, the drawback is that learning good surrogates from experiments typically requires a large, representative portfolio of long-run experiments that few platforms possess. We conclude that there is no substitute for a well-run long-term experiment, whether for learning surrogates or validating them, and we highlight open challenges including evolving treatments, persistent treatments not fully mediated by short-term proxies, and mismatch between experimental samples and the target population.

In the presence of interference, where the treatment assigned to one unit can affect the outcomes of others, many causal estimands depend on the treatment-assignment policy under which the experiment is conducted. This policy dependence creates a fundamental challenge for off-policy estimation, where the goal is to estimate causal quantities under a hypothetical intervention policy different from the one used to collect data. We study this problem of off-policy estimation of causal effects for heterogeneous Bernoulli policies. By representing exposure-weighted potential outcomes in the biased Fourier basis of the experimental design, we construct, for any prespecified Fourier subspace encoding the assumed interference structure, the unique minimum-L^2 weight that transports every function in that subspace. Global and local inverse-probability weights, linear-interference weights, and no-interference weights are special cases. The weight variance is a structured chi-square distance between the experiment and target policies. When the assumed interference structure is misspecified, the introduced bias couples the omitted outcome spectrum with the corresponding policy-shift coefficients, yielding a sharp robustness bound and a bias-variance trade-off. A Fourier-neighborhood-overlap condition gives consistency under structured interference, and we state a Doob-martingale central limit theorem for off-policy estimators. As the variance is not identified, we derive identifiable bounds and associated conservative estimators of the variance. Simulations illustrate these theoretical results for the design and analysis of experiments under network interference and design mismatch.

Empirical social networks are characterized by a high degree of triadic closure (i.e., transitivity, clustering), whereby network neighbors of the same individual are also likely to be directly connected. It is unknown to what degree this results from dispositions to form such ties (i.e., to close open triangles) per se or from other processes, such as homophily and more opportunities for exposure. These are difficult to disentangle in many settings, but in social media not only can they be decomposed, but platforms frequently make decisions that depend on these distinct processes. Here, using a field experiment on social media, we randomize the existing network structure that a user faces when followed by a target account that we control, and we examine whether they reciprocate this tie formation. Being randomly assigned to have an existing tie to an account that follows the target user increases tie formation by 35%. Through the use of multiple control conditions in which the relevant tie is absent (never existent or removed), we attribute this effect specifically to a minimal cue that indicates the presence of a potential mutual follower. Theory suggests that triadic closure should be especially likely in open triads of strong ties, and we find larger effects when the subject has interacted more with the existing follower. These results indicate a substantial role for tendencies toward triadic closure, but one that is substantially smaller than what might be inferred from prior observational studies. Platforms and others may rely on these tendencies in encouraging tie formation, with broader implications for network structure and information diffusion in online networks.

In settings where units' outcomes are affected by others' treatments, there has been a proliferation of ways to quantify effects of treatments on outcomes. Here we describe how many proposed estimands can be represented as involving one of two ways of averaging over units and treatment assignments. The more common representation often results in quantities that are irrelevant, or at least insufficient, for optimal choice of policies governing treatment assignment. The other representation often yields quantities that lack an interpretation as summaries of unit-level effects, but that we argue may still be relevant to policy choice. Among various estimands, the expected average outcome – or its contrast between two different policies – can be represented both ways and, we argue, merits further attention.

Social corrections – where users correct each other – can help rectify inaccurate beliefs. However, social corrections are often ignored. Here we ask under what conditions social corrections promote engagement from corrected users, allowing for greater insight into how users respond to debunking messages (even if such responses are negative). Prior work suggests two key factors may help promote engagement with corrections – partisan alignment between users, and social connections between users. We investigate these factors here. First, we conducted a field experiment on Twitter (X) using human-looking bots to examine how shared partisanship and prior social connection affect correction engagement. We randomized whether our accounts identified as Democrat or Republican, and whether they followed Twitter users and liked three of their tweets before correcting them (creating a minimal social connection). We found that shared partisanship had no significant effect in the baseline (no social connection) condition. Interestingly, social connection increased engagement with corrections from co-partisans. Effects in the social counter-partisan condition were ambiguous. Follow-up survey experiments largely replicated these results and found evidence for a generalized norm of responding, wherein people feel more obligated to respond to people who follow them – even outside the context of misinformation correction. Our findings have important implications for increasing engagement with social corrections online.

Regression discontinuity designs are used to estimate causal effects in settings where treatment is determined by whether an observed running variable crosses a pre-specified threshold. While the resulting sampling design is sometimes described as akin to a locally randomized experiment in a neighborhood of the threshold, standard formal analyses do not make reference to probabilistic treatment assignment and instead identify treatment effects via continuity arguments. Here we propose a new approach to identification, estimation, and inference in regression discontinuity designs that exploits measurement error in the running variable. Under an assumption that the measurement error is exogenous, we show how to consistently estimate causal effects using a class of linear estimators that weight treated and control units so as to balance a latent variable of which the running variable is a noisy measure. We find this approach to facilitate identification of both familiar estimands from the literature, as well as policy-relevant estimands that correspond to the effects of realistic changes to the existing treatment assignment rule. We demonstrate the method with a study of retention of HIV patients and evaluate its performance using simulated data and a regression discontinuity design artificially constructed from test scores in early childhood.

Interventions to reduce misinformation sharing have been a major focus in recent years. Developing “content-neutral” interventions that do not require specific fact-checks or warnings related to individual false claims is particularly important in developing scalable solutions. Here, we provide the first evaluations of a content-neutral intervention to reduce misinformation sharing conducted at scale in the field. Specifically, across two on-platform randomized controlled trials, one on Meta’s Facebook (N=33,043,471) and the other on Twitter (N=75,763), we find that simple messages reminding people to think about accuracy—delivered to large numbers of users using digital advertisements—reduce misinformation sharing, with effect sizes on par with what is typically observed in digital advertising experiments. On Facebook, in the hour after receiving an accuracy prompt ad, we found a 2.6% reduction in the probability of being a misinformation sharer among users who had shared misinformation the week prior to the experiment. On Twitter, over more than a week of receiving 3 accuracy prompt ads per day, we similarly found a 3.7% to 6.3% decrease in the probability of sharing low-quality content among active users who shared misinformation pre-treatment. These findings suggest that content-neutral interventions that prompt users to consider accuracy have the potential to complement existing content-specific interventions in reducing the spread of misinformation online.

Long ties that bridge socially separate regions of networks are critical for the spread of contagions, such as innovations or adoptions of new norms. Contrary to previous thinking, long ties have now been found to accelerate social contagions, even for behaviours that involve the social reinforcement of adoption by network neighbours.

Decision makers often want to target interventions so as to maximize an outcome that is observed only in the long term. This typically requires delaying decisions until the outcome is observed or relying on simple short-term proxies for the long-term outcome. Here, we build on the statistical surrogacy and policy learning literatures to impute the missing long-term outcomes and then approximate the optimal targeting policy on the imputed outcomes via a doubly robust approach. We first show that conditions for the validity of average treatment effect estimation with imputed outcomes are also sufficient for valid policy evaluation and optimization; furthermore, these conditions can be somewhat relaxed for policy optimization. We apply our approach in two large-scale proactive churn management experiments at The Boston Globe by targeting optimal discounts to its digital subscribers with the aim of maximizing long-term revenue. Using the first experiment, we evaluate this approach empirically by comparing the policy learned using imputed outcomes with a policy learned on the ground-truth, long-term outcomes. The performance of these two policies is statistically indistinguishable, and we rule out large losses from relying on surrogates. Our approach also outperforms a policy learned on short-term proxies for the long-term outcome. In a second field experiment, we implement the optimal targeting policy with additional randomized exploration, which allows us to update the optimal policy for future subscribers. Over three years, our approach had a net-positive revenue impact in the range of $4–$5 million compared with the status quo. This paper was accepted by Eric Anderson, marketing. Funding: This work was supported by Boston Globe Media. Supplemental Material: The online appendix and data are available at https://doi.org/10.1287/mnsc.2023.4881 .

In widely used models of biological contagion, interventions that randomly rewire edges (generally making them 'longer') accelerate spread. However, recent work has argued that highly clustered, rather than random, networks facilitate the spread of threshold-based contagions, such as those motivated by myopic best response for adoption of new innovations, norms and products in games of strategic complement. Here we show that minor modifications to this model reverse this result, thereby harmonizing qualitative facts about how network structure affects contagion. We analyse the rate of spread over circular lattices with rewired edges and show that having a small probability of adoption below the threshold probability is enough to ensure that random rewiring accelerates the spread of a noisy threshold-based contagion. This conclusion is verified in simulations of empirical networks and remains valid with partial but frequent enough rewiring and when adoption decisions are reversible but infrequently so, as well as in high-dimensional lattice structures.

In addition to more personalized content feeds, some leading social media platforms give a prominent role to content that is more widely popular. On Twitter, "trending topics" identify popular topics of conversation on the platform, thereby promoting popular content which users might not have otherwise seen through their network. Hence, "trending topics" potentially play important roles in influencing the topics users engage with on a particular day. Using two carefully constructed data sets from India and Turkey, we study the effects of a hashtag appearing on the trending topics page on the number of tweets produced with that hashtag. We specifically aim to answer the question: How many new tweeting using that hashtag appear because a hashtag is labeled as trending? We distinguish the effects of the trending topics page from network exposure and find there is a statistically significant, but modest, return to a hashtag being featured on trending topics. Analysis of the types of users impacted by trending topics shows that the feature helps less popular and new users to discover and spread content outside their network, which they otherwise might not have been able to do.

Social networks shape and reflect economic life. Prior studies have identified long ties, which connect people who lack mutual contacts, as a correlate of individuals’ success within firms and places’ economic prosperity. However, we lack population-scale evidence of the individual-level link between long ties and economic prosperity, and why some people have more long ties remains obscure. Here, using a social network constructed from interactions on Facebook, we establish a robust association between long ties and economic outcomes and study disruptive life events hypothesized to cause formation of long ties. Consistent with prior aggregated results, administrative units with a higher fraction of long ties tend to have higher-income and economic mobility. Individuals with more long ties live in higher-income places and have higher values of proxies for economic prosperity (e.g., using more Internet-connected devices and making more donations). Furthermore, having stronger long ties (i.e., with higher intensity of interaction) is associated with better outcomes, consistent with an advantage from the structural diversity constituted by long ties, rather than them being weak ties per se. We then study the role of disruptive life events in the formation of long ties. Individuals who have migrated between US states, have transferred between high schools, or have attended college out-of-state have a higher fraction of long ties among their contacts many years after the event. Overall, these results suggest that long ties are robustly associated with economic prosperity and highlight roles for important life experiences in developing and maintaining long ties.

Artificial Intelligence (AI) automates human decisions. Algorithmic pricing, a form of AI, sets prices by a computer. It is now common currency in ride-hailing, travel, drugs, gasoline, online goods---And great price variability characterizes all those settings. However, little is known about how consumers respond to encountering frequently changing prices. This paper uses clickstream data from an online retailer in the U.S. that varied pricing methods to examine effects of frequently-changing prices on purchase behavior. The evidence shows that exposure to price variability exacerbates price sensitivity. These findings are confirmed in online lab experiments. Additionally, an underlying mechanism is price salience.

Despite the availability of multiple safe vaccines, vaccine hesitancy may present a challenge to successful control of the COVID-19 pandemic. As with many human behaviors, people’s vaccine acceptance may be affected by their beliefs about whether others will accept a vaccine (i.e., descriptive norms). However, information about these descriptive norms may have different effects depending on the actual descriptive norm, people’s baseline beliefs, and the relative importance of conformity, social learning, and free-riding. Here, using a pre-registered, randomized experiment ( N = 484,239) embedded in an international survey (23 countries), we show that accurate information about descriptive norms can increase intentions to accept a vaccine for COVID-19. We find mixed evidence that information on descriptive norms impacts mask wearing intentions and no statistically significant evidence that it impacts intentions to physically distance. The effects on vaccination intentions are largely consistent across the 23 included countries, but are concentrated among people who were otherwise uncertain about accepting a vaccine. Providing normative information in vaccine communications partially corrects individuals’ underestimation of how many other people will accept a vaccine. These results suggest that presenting people with information about the widespread and growing acceptance of COVID-19 vaccines helps to increase vaccination intentions.

Seeding the most influential individuals based on the contact structure can substantially enhance the extent of a spread over the social network. Most of the influence maximization literature assumes the knowledge of the entire network graph. However, in practice, obtaining full knowledge of the network structure is very costly. We propose polynomial-time algorithms that provide almost tight approximation guarantees using a bounded number of queries to the graph structure. We also provide impossibility results to lower bound the query complexity and show tightness of our guarantees.

Social networks affect the diffusion of information, and thus have the potential to reduce or amplify inequality in access to opportunity. We show empirically that social networks often exhibit a much larger potential for unequal diffusion across groups along paths of length 2 and 3 than expected by our random graph models. We argue that homophily alone cannot not fully explain the extent of unequal diffusion and attribute this mismatch to unequal distribution of cross-group links among the nodes. Based on this insight, we develop a variant of the stochastic block model that incorporates the heterogeneity in cross-group linking. The model provides an unbiased and consistent estimate of assortativity or homophily on paths of length 2 and provide a more accurate estimate along paths of length 3 than existing models. We characterize the null distribution of its log-likelihood ratio test and argue that the goodness of fit test is valid only when the network is dense. Based on our empirical observations and modeling results, we conclude that the impact of any departure from equal distribution of links to source nodes in the diffusion process is not limited to its first order effects as some nodes will have fewer direct links to the sources. More importantly, this unequal distribution will also lead to second order effects as the whole group will have fewer diffusion paths to the sources.

1. At established platforms, algorithmic ranking and recommendation involve using many signals and are typically not aimed at simply maximizing short-run engagement.2. Quantifying the impacts of algorithmic ranking is quite difficult,even with access to proprietary data. This is not only becauseof the complexity of these technical systems, but due to people’scomplex and often strategic responses to changes in algorithms.3. We lack clear evidence about broader benefits or harms of algorithmic ranking. Nonetheless, simple rankings and recommendations (e.g., chronological, overall popularity) can make some formsof undesirable strategic behavior easier.4. Policy-makers can protect the ability of external researchers toprobe these systems, and they can provide clear paths for platforms to retain and share data in privacy-preserving ways.

Free AccessAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked InEmail Go to SectionFree Access HomeINFORMS Journal on Data ScienceVol. 1, No. 1 Commentary on “Causal Decision Making and Causal Effect Estimation Are Not the Same…and Why It Matters”: On Loss Functions and Bias–Variance Tradeoffs in Causal Estimation and DecisionsDean Eckles Dean Eckles Published Online:16 Mar 2022https://doi.org/10.1287/ijds.2022.0012"Commentary on “Causal Decision Making and Causal Effect Estimation Are Not the Same…and Why It Matters”: On Loss Functions and Bias–Variance Tradeoffs in Causal Estimation and Decisions." INFORMS Journal on Data Science, 1(1), pp. 17–18 Previous Back to Top Next FiguresReferencesRelatedInformation Volume 1, Issue 1April-June 2022Pages 1-113, C2 Article Information Metrics Information Received:November 02, 2021Accepted:November 14, 2021Published Online:March 16, 2022 Copyright © 2022, INFORMSCite asDean Eckles (2022) Commentary on “Causal Decision Making and Causal Effect Estimation Are Not the Same…and Why It Matters”: On Loss Functions and Bias–Variance Tradeoffs in Causal Estimation and Decisions. INFORMS Journal on Data Science 1(1):17-18. https://doi.org/10.1287/ijds.2022.0012 Keywordscausal inferencebias–variance tradeoffpolicy learningPDF download

When trying to maximize the adoption of a behavior in a population connected by a social network, it is common to strategize about where in the network to seed the behavior, often with an element of randomness. Selecting seeds uniformly at random is a basic but compelling strategy in that it distributes seeds broadly throughout the network. A more sophisticated stochastic strategy, one-hop targeting, is to select random network neighbors of random individuals; this exploits a version of the friendship paradox, whereby the friend of a random individual is expected to have more friends than a random individual, with the hope that seeding a behavior at more connected individuals leads to more adoption. Many seeding strategies have been proposed, but empirical evaluations have demanded large field experiments designed specifically for this purpose and have yielded relatively imprecise comparisons of strategies. Here we show how stochastic seeding strategies can be evaluated more efficiently in such experiments, how they can be evaluated “off-policy” using existing data arising from experiments designed for other purposes, and how to design more efficient experiments. In particular, we consider contrasts between stochastic seeding strategies and analyze nonparametric estimators adapted from policy evaluation and importance sampling. We use simulations on real networks to show that the proposed estimators and designs can substantially increase precision while yielding valid inference. We then apply our proposed estimators to two field experiments, one that assigned households to an intensive marketing intervention and one that assigned students to an antibullying intervention. This paper was accepted by Gui Liberali, Management Science Special Section on Data-Driven Prescriptive Analytics.

Social networks play a predominant role in determining how information spreads between individuals. Previous works suggest that long ties, which connect people who do not share any mutual contact, provide access to valuable information on economic opportunities. However, no population-scale study has determined how long ties relate to economic outcomes and how such ties are formed. Using a novel dataset from Facebook, we reconstruct the network of interactions between users and we uncover a strong relationship between the share of long ties and economic outcomes at the local level in the United States and in Mexico. Administrative units with a higher proportion of long ties have higher incomes, higher economic mobility, lower unemployment rates and higher wealth, even after adjusting for potential confounders of these outcomes. In contrast to the weak tie theory, we find that having stronger long ties is associated with better economic outcomes. Furthermore, we discover that users with a higher proportion of long ties are more likely to have migrated between US states, to have transferred to a different high school, and to have attended college outside of their home state. Taken together, these results suggest that long ties contribute to economic prosperity and highlight the role played by disruptive life events in the formation of these ties.