Online controlled experiments, or A/B tests, are commonly used to evaluate ideas and their implementations in websites and applications (G. Z. Georgiev 2019; Kohavi, Tang, and Xu 2020; Luca and Bazer man 2020; Thomke 2020; Weigel 2024). Large companies run tens of thousands of A/B tests per year. The thirteen organizations that attended the Practical Online Controlled Experiments Summit in 2018 "tested more than one hundred thousand experiment treatments last year"

The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.

A/B tests, or online controlled experiments, are used heavily in the software industry to evaluate implementations of ideas, as the paradigm is the gold standard in science for establishing causality: the changes introduced in the treatment caused the changes to the metrics of interest with high probability. What distinguishes software experiments, or A/B tests, from experiments in many other domains is the scale (e.g., over 100 experiment treatments may launch on a given workday in large companies) and the effect sizes that matter to the business are small (e.g., a 3% improvement to conversion rate from a single experiment is a cause for celebration). The humbling reality is that most experiments fail to improve key metrics, and success rates of only about 10-20% are most common. With low success rates, the industry standard alpha threshold of 0.05 implies a high probability of false positives. We begin with motivation about why false positives are expensive in many software domains. We then offer several approaches to estimate the true success rate of experiments, given the observed "win" rate (statistically significant positive improvements), and show examples from Expedia and Optimizely. We offer a modified procedure for experimentation, based in sequential group testing, that selectively extends experiments to reduce false positives, increase power, at a small increase to runtime. We conclude with a discussion of the difference between ideas and experiments in practice, terms that are often incorrectly used interchangeably.

A/B tests, or online controlled experiments, are heavily used in industry to evaluate implementations of ideas. While the statistics behind controlled experiments are well documented and some basic pitfalls known, we have observed some seemingly intuitive concepts being touted, including by A/B tool vendors and agencies, which are misleading, often badly so. Our goal is to describe these misunderstandings, the "intuition" behind them, and to explain and bust that intuition with solid statistical reasoning. We provide recommendations that experimentation platform designers can implement to make it harder for experimenters to make these intuitive mistakes.

A/B tests, or online controlled experiments, are heavily used in industry to evaluate implementations of ideas. While the statistics behind controlled experiments are well documented and some basic pitfalls known, we have observed some seemingly intuitive concepts being touted, including by A/B tool vendors and agencies, which are misleading, often badly so. Our goal is to describe these misunderstandings, the "intuition" behind them, and to explain and bust that intuition with solid statistical reasoning. We provide recommendations that experimentation platform designers can implement to make it harder for experimenters to make these intuitive mistakes.

The process of feature selection is generally used to minimize the size of dataset, to overcome the problem of over fitting and to increase the classifier efficiency. We proposed the JMIM i. e. Joint Mutual Information Maximization algorithm to extract feature and for creation of feature subset efficiently. These algorithms are based on joint mutual information. It follows maximum of minimum strategy. In this paper our aim is to work on utilization of JMIM algorithm, then we compare upcoming outcome with the previously highlighted problems in existed feature selection system. In utilization of JMIM algorithm, we are expecting that our simultaneous processing of feature set selection process will reduces time required for overall execution. As a part of our contribution the process distributed over different clouds that helps in execution and triggers the process.

Why you care: To design and run a good online controlled experiment, you need metrics that meet certain characteristics. They must be measurable in the short term (experiment duration) and computable, as well as sufficiently sensitive and timely to be useful for experimentation. If you use multiple metrics to measure success for an experiment, ideally you may want to combine them into an Overall Evaluation Criterion (OEC), which is believed to causally impact long-term objectives. It often requires multiple iterations to adjust and refine the OEC, but as the quotation above, by Eliyahu Goldratt, highlights, it provides a clear alignment mechanism to the organization.

Why you care: The choice of randomization unit is critical in experiment design, as it affects both the user experience as well as what metrics can be used in measuring the impact of an experiment. When building an experimentation system, you need to think through what options you want to make available. Understanding the options and the considerations to use when choosing amongst them will lead to improved experiment design and analysis.

A summary is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.

A summary is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.

Why you care: Triggering provides experimenters with a way to improve sensitivity (statistical power) by filtering out noise created by users who could not have been impacted by the experiment. As organizational experimentation maturity improves, we see more triggered experiments being run.

The joint analysis of species’ evolutionary relatedness and their morphological evolution has 2 offered much promise in understanding the processes that underpin the generation of biological 3 diversity. Disparity through time (DTT) is a popular method that estimates the relative trait 4 disparity within and between subclades at each time point, and compares this to the null hypothesis 5 that trait values follow an uncorrelated random walk along the time calibrated phylogenetic tree. A 6 simulation envelope is normally created by calculating, at every time point, the 95% minimum and 7 95% maximum disparity values from multiple simulations of the null model on the phylogenetic 8 tree. The null hypothesis is rejected whenever the empirical DTT curve falls outside of this 9 envelope, and these time periods may then be linked to events that may have sparked non-random 10 trait evolution. However, this method of envelope construction leads to multiple testing and a poor, 11 uncontrolled, false positive rate. As a consequence it cannot be recommended. A recently 12 developed method in spatial statistics is introduced that constructs a confidence envelope by giving 13 each DTT curve a single ranking value based upon its most extreme disparity value. This method 14 avoids the pitfalls of multiple testing whilst retaining a visual interpretation. Results using 15 simulated data show this new test has desirable type 1 properties and is at least as powerful in 16 correctly rejecting the null hypothesis as the morphological disparity index and node height test that 17 lack a visual interpretation. Three example datasets are reanalyzed to show how the new test may 18 lead to different inferences being drawn. Overall the results suggest the new rank envelope test 19 should be used in null model testing for DTT analyses, and that there is no need to combine the 20 envelope test with other tests such as has been done previously. Moreover, the rank envelope 21 method can easily be adopted into recently developed posterior predictive simulation methods. More generally, the rank envelope test should be adopted when-ever a null model produces a vector 23 of correlated values and the user wants to determine where the empirical data is different to the null 24 model.

Why you care: Guardrail metrics are critical metrics designed to alert experimenters about violated assumptions. There are two types of guardrail metrics: organizational and trust-related. Chapter 7 discusses organizational guardrails that are used to protect the business, and this chapter describes the Sample Ratio Mismatch (SRM) in detail, which is a trust-related guardrail. The SRM guardrail should be included for every experiment, as it is used to ensure the internal validity and trustworthiness of the experiment results. A few other trust-related guardrail metrics are also described here.