AB testing evaluates the difference between a control and a treatment in a statistically rigorous manner. Continuous monitoring allows statistical evaluation of an AB test as it proceeds. One goal of continuous monitoring is early stopping – confirming a statistically significant difference between control and treatment as soon as possible. Another goal is to maintain some statistical capability to discover significant differences later in the test if they cannot be confirmed earlier. These goals are in conflict – looser requirements for early stopping leave us with more stringent ones for later. This paper shows that it is impossible to maintain a constant requirement for significance for tests that have no a priori stopping time, but we can come arbitrarily close to that goal by using tests that require repeated significant results to con rm statistically significant differences between treatment and control.

One method to offer some bidders a discount in a first-price auction is to augment their bids when selecting a winner but only charge them their original bids should they win. Another method is to use their original bids to select a winner, then charge them a discounted price that is lower than their bid should they win. We show that the two methods have equivalent auction outcomes, for equal additive discounts and for multiplicative ones with appropriate adjustments to discount amounts. As a result, they have corresponding equilibria when equilibria exist. We also show that with the same level of multiplicative adjustments, bidders with discounts should prefer an augmented bid to a discounted price. Then we estimate optimal bid functions for valuation distributions based on data from online advertising auctions, and show how different discount levels affect auction outcomes for those bid functions.

For a bucket test with a single criterion for success and a fixed number of samples or testing period, requiring a p-value less than a specified value of α for the success criterion produces statistical confidence at level 1 - α. For multiple criteria, a Bonferroni correction that partitions α among the criteria produces statistical confidence, at the cost of requiring lower p-values for each criterion. The same concept can be applied to decisions about early stopping, but that can lead to strict requirements for p-values. We show how to address that challenge by requiring criteria to be successful at multiple decision points.

The k-Minimum Values () data sketch algorithm stores the k least hash keys generated by hashing the items in a dataset. We show that compression based on ordering the keys and encoding successive differences can offer O(log n) bits per key in expected storage savings, where n is the number of unique values in the data set. We also show that O(log n) expected bits saved per key is optimal for any form of compression for the k least of n random values – that the encoding method is near-optimal among all methods to encode a sketch. We present a practical method to perform that compression, show that it is computationally efficient, and demonstrate that its average savings in practice is within about five percent of the theoretical minimum based on entropy. We verify that our method outperforms off-the-shelf compression methods, and we demonstrate that it is practical, using real and synthetic data.

Generative AI uses a large set of sources to create content. The content generated by large language models is text. Often, that text contains statements that are inaccurate or false, sometimes called hallucinations. We explore how identifying citations for the generated text can enable people to determine whether to trust the statements in the text, by allowing different users to specify different trusted sets of sources as candidates for citations. Then we propose methods to eliminate or correct untrustworthy statements. We also consider how citations can help build consensus among people who have different trusted sources of information, by using a large language model to construct text, then editing the text so that it is supported by citations drawn from multiple sets of trusted sources. By using generative AI as a go-between, such a process may allow parties with mutual distrust to discover and confirm areas of agreement. This paper is a proposal for systems that enhance large language models’ usefulness and an outline of some challenges and methods for such systems; it is not a record of system development or testing.

In this paper, we develop a non-asymptotic local normal approximation for multinomial probabilities. First, we use it to find non-asymptotic total variation bounds between the measures induced by uniformly jittered multinomials and the multivariate normals with the same means and covariances. From the total variation bounds, we also derive a comparison of the cumulative distribution functions and quantile coupling inequalities between Pearson's chi-square statistic (written as the normalized quadratic form of a multinomial vector) and its multivariate normal analogue. We apply our results to find confidence intervals for the negative entropy of discrete distributions. Our method can be applied more generally to find confidence intervals for strictly convex functions of the weights of discrete distributions.

Data sketching has emerged as a key infrastructure for large-scale data analysis on streaming and distributed data. Merging sketches enables efficient estimation of cardinalities and frequency histograms over distributed data. However, merging sketches can require that each sketch stores hash codes for identifiers in different data sets or partitions, in order to perform effective matching. This can reveal identifiers during merging or across different data set or partition owners. This paper presents a framework to use noisy hash codes, with the noise level selected to obfuscate identifiers while allowing matching, with high probability. We give probabilistic error bounds on simultaneous obfuscation and matching, concluding that this is a viable approach.

Big data is often used to generate reports consisting of computed statistics. When the statistics are displayed, they may be marked for significance based on p-values. For multiple statistics, we propose a system to mark for significance adjusted for the number of statistics, using a Bonferroni correction in general, and a slightly milder correction for independent statistics.

if a data set covers a population of interest then statistics can be computed exactly. If it is a sample of the population (or distribution) of interest then statistics must be estimated. We consider conditions for those estimates to be accurate or inaccurate.

We show how recent technologies could be combined to enable contactless, secure, asynchronous, and convenient food delivery, by solving the last-mile/last-hour problem with a food drop-off box that keeps delivered foods at their desired temperatures until the recipient can collect them. Supporting technologies include internet-of-things, wireless communications, wireless access control, and UV sanitization. We also discuss how the device would interact with autonomous vehicles.

We introduce methods to bound the mean of a discrete distribution (or finite population) based on sample data, for random variables with a known set of possible values. In particular, the methods can be applied to categorical data with known category-based values. For small sample sizes, we show how to leverage the knowledge of the set of possible values to compute bounds that are stronger than for general random variables such as standard concentration inequalities.

One way to estimate a statistic over a large data set is to draw a sample consisting of some records from the data set, and compute the statistic over the sample as an estimate of the statistic over the data set. This procedure may fail to produce an accurate estimate. Using one sample for multiple statistics reduces computation and latency, but it can increase the probability of multiple failures to produce accurate estimates, because estimates based on the same sample may not have independent failure probabilities. We show how to bound the probability of multiple failures for sequences of estimates over one or more samples.

For a voting ensemble that selects an odd-sized subset of the ensemble classifiers at random for each example, applies them to the example, and returns the majority vote, we show that any number of voters may minimize the error rate over an out-of-sample distribution. The optimal number of voters depends on the out-of-sample distribution of the number of classifiers in error. To select a number of voters to use, estimating that distribution then inferring error rates for numbers of voters gives lower-variance estimates than directly estimating those error rates.

Today, web-based companies use user data to provide and enhance services to users, both individually and collectively. Some also analyze user data for other purposes, for example to select advertisements or price offers for users. Some even use or allow the data to be used to evaluate investments in financial markets. Users' concerns about how their data is or may be used has prompted legislative action in the European Union and congressional questioning in the United States. But data can also benefit society, for example giving early warnings for disease outbreaks, allowing in-depth study of relationships between genetics and disease, and elucidating local and macroeconomic trends in a timely manner. So, instead of just a focus on privacy, in the future, users may insist that their data be used on their behalf. We explore potential frameworks for groups of consenting, informed users to pool their data for their own benefit and that of society, discussing directions, challenges, and evolution for such efforts.

We propose a system for privacy-aware machine learning. The data provider encodes each record in way that avoids revealing information about the record's field values or about the ordering of values from different records. A service provider stores the encoded records and uses them to perform classification on queries consisting of encoded input field values. The encoding provides privacy for the data provider from the service provider and from a third party issuing unauthorized queries. But the encoding makes regression-based and many tree-based classifiers impossible to implement. It does allow histogram-type classifiers that are based on category membership, and we present one such classification method that ensures data sufficiency on a per-classification basis.

In a second-price auction with i.i.d. (independent identically distributed) bidder valuations, adding bidders increases expected buyer surplus if the distribution of valuations has a sufficiently heavy right tail. While this does not imply that a bidder in an auction should prefer for more bidders to join the auction, it does imply that a bidder should prefer it in exchange for the bidder being allowed to participate in more auctions. Also, for a heavy-tailed valuation distribution, marginal expected seller revenue per added bidder remains strong even when there are already many bidders.

A data sketch algorithm scans a big data set, collecting a small amount of data -- the sketch, which can be used to statistically infer properties of the big data set. Some data sketch algorithms take a fixed-size random sample of a big data set, and use that sample to infer frequencies of items that meet various criteria in the big data set. This paper shows how to statistically infer probably approximately correct (PAC) bounds for those frequencies, efficiently, and precisely enough that the frequency bounds are either sharp or off by only one, which is the best possible result without exact computation.

Quality data is a fundamental contributor to success in statistics and machine learning. If a statistical assessment or machine learning leads to decisions that create value, data contributors may want a share of that value. This paper presents methods to assess the value of individual data samples, and of sets of samples, to apportion value among different data contributors. We use Shapley values for individual samples and Owen values for combined samples, and show that these values can be computed in polynomial time in spite of their definitions having numbers of terms that are exponential in the number of samples.

This note describes how to collect charges for ad impact on user experience. The charge may be per-view, to account for impact on user experience from viewing an ad, or per-click, to account for impact from clicking on the ad. The results for per-click charges also apply to per-conversion charges or per-action charges. Conceivably, a marketplace could assess both kinds of charges.

We introduce the speculate-correct method to derive error bounds for local classifiers. Using it, we show that k-nearest neighbor classifiers, in spite of their famously fractured decision boundaries, have exponential error bounds with $$\hbox {O} \left( \sqrt{(k + \ln n)/n} \right) $$ range around an estimate of generalization error for n in-sample examples.