The prevalence and low cost of LLMs have led to a rise of synthetic content. From review sites to court documents, “natural” content has been contaminated by data points that appear similar to natural data, but are in fact LLM-generated. In this work we revisit fundamental learning theory questions in this, now ubiquitous, setting. We model this scenario as a sequence of learning tasks where the input is a mix of natural and synthetic data, and the learning algorithms are oblivious to the origin of any individual example. We study the possibilities and limitations of ERM in this setting. For the problem of estimating the mean of an arbitrary d-dimensional distribution, we find that while ERM converges to the true mean, it is outperformed by an algorithm that assigns non-uniform weights to examples from different generations of data. For the PAC learning setting, the disparity is even more stark. We find that ERM does not always converge to the true concept, echoing the model collapse literature. However, we show there are algorithms capable of learning the correct hypothesis for arbitrary VC classes and arbitrary amounts of contamination.

Synthetically-generated data plays an increasingly larger role in training large language models. However, while synthetic data has been found to be useful, studies have also shown that without proper curation it can cause LLM performance to plateau, or even "collapse", after many training iterations. In this paper, we formalize this question and develop a theoretical framework to investigate how much curation is needed in order to ensure that LLM performance continually improves. Our analysis is inspired by boosting, a classic machine learning technique that leverages a very weak learning algorithm to produce an arbitrarily good classifier. The approach we analyze subsumes many recently proposed methods for training LLMs on synthetic data, and thus our analysis sheds light on why they are successful, and also suggests opportunities for future improvement. We present experiments that validate our theory, and show that dynamically focusing labeling resources on the most challenging examples --- in much the same way that boosting focuses the efforts of the weak learner --- leads to improved performance.

Differentially private (DP) language model inference is an approach for generating private synthetic text. A sensitive input example is used to prompt an off-the-shelf large language model (LLM) to produce a similar example. Multiple examples can be aggregated together to formally satisfy the DP guarantee. Prior work creates inference batches by sampling sensitive inputs uniformly at random. We show that uniform sampling degrades the quality of privately generated text, especially when the sensitive examples concern heterogeneous topics. We remedy this problem by clustering the input data before selecting inference batches. Next, we observe that clustering also leads to more similar next-token predictions across inferences. We use this insight to introduce a new algorithm that aggregates next token statistics by privately computing medians instead of averages. This approach leverages the fact that the median has decreased local sensitivity when next token predictions are similar, allowing us to state a data-dependent and ex-post DP guarantee about the privacy properties of this algorithm. Finally, we demonstrate improvements in terms of representativeness metrics (e.g., MAUVE) as well as downstream task performance. We show that our method produces high-quality synthetic data at significantly lower privacy cost than a previous state-of-the-art method.

Modern Web APIs allow developers to provide extensively customized experiences for website visitors, but the richness of the device information they provide also make them vulnerable to being abused to construct browser fingerprints, device-specific identifiers that enable covert tracking of users even when cookies are disabled. Previous research has established entropy, a measure of information, as the key metric for quantifying fingerprinting risk. However, earlier studies had two major limitations. First, their entropy estimates were based on either a single website or a very small sample of devices. Second, they did not adequately consider correlations among different Web APIs, potentially grossly overestimating their fingerprinting risk. We provide the first study of browser fingerprinting which addresses the limitations of prior work. Our study is based on actual visited pages and Web APIs reported by tens of millions of real Chrome browsers in-the-wild. We accounted for the dependencies and correlations among Web APIs, which is crucial for obtaining more realistic entropy estimates. We also developed a novel experimental design that accurately and efficiently estimates entropy while never observing too much information from any single user. Our results provide an understanding of the distribution of entropy for different website categories, confirm the utility of entropy as a fingerprinting proxy, and offer a method for evaluating browser enhancements which are intended to mitigate fingerprinting.

We present an approach for generating differentially private synthetic text using large language models (LLMs), via private prediction. In the private prediction framework, we only require the output synthetic data to satisfy differential privacy guarantees. This is in contrast to approaches that train a generative model on potentially sensitive user-supplied source data and seek to ensure the model itself is safe to release. We prompt a pretrained LLM with source data, but ensure that next-token predictions are made with differential privacy guarantees. Previous work in this paradigm reported generating a small number of examples (<10) at reasonable privacy levels, an amount of data that is useful only for downstream in-context learning or prompting. In contrast, we make changes that allow us to generate thousands of high-quality synthetic data points, greatly expanding the set of potential applications. Our improvements come from an improved privacy analysis and a better private selection mechanism, which makes use of the equivalence between the softmax layer for sampling tokens in LLMs and the exponential mechanism. Furthermore, we introduce a novel use of public predictions via the sparse vector technique, in which we do not pay privacy costs for tokens that are predictable without sensitive data; we find this to be particularly effective for structured data.

Differentially private training algorithms like DP-SGD protect sensitive training data by ensuring that trained models do not reveal private information. An alternative approach, which this paper studies, is to use a sensitive dataset to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data. Doing so has several advantages: synthetic data can be reused for other tasks (including for hyper parameter tuning), retained indefinitely, and shared with third parties without sacrificing privacy. However, generating private synthetic data is much harder than training a private model. To improve performance on text data, recent work has utilized public data by starting with a pre-trained generative language model and privately fine-tuning it on sensitive data. This model can be used to sample a DP synthetic dataset. While this strategy seems straightforward, executing it has proven problematic. Previous approaches either show significant performance loss, or have, as we show, critical design flaws. In this paper we demonstrate that a proper training objective along with tuning fewer parameters results in excellent DP synthetic data quality. Our approach is competitive with direct DP-training of downstream classifiers in terms of performance on downstream tasks. Further, we demonstrate that our DP synthetic data is not only useful for downstream classifier training, but also to tune those same models.

Browser fingerprinting can be used to identify and track users across the Web, even without cookies, by collecting attributes from users' devices to create unique 'fingerprints'. This technique and resulting privacy risks have been studied for over a decade. Yet further research is limited because prior studies used data not publicly available. Additionally, data in prior studies lacked user demographics. Here we provide a first-of-its-kind dataset to enable further research. It includes browser attributes with users' demographics and survey responses, collected with informed consent from 8,400 US study participants. We use this dataset to demonstrate how fingerprinting risks differ across demographic groups. For example, we find lower income users are more at risk, and find that as users' age increases, they are both more likely to be concerned about fingerprinting and at real risk of fingerprinting. Furthermore, we demonstrate an overlooked risk: user demographics, such as gender, age, income level and race, can be inferred from browser attributes commonly used for fingerprinting, and we identify which browser attributes most contribute to this risk. Our data collection process also conducted an experiment to study what impacts users' likelihood to share browser data for open research, in order to inform future data collection efforts, with responses from 12,461 total participants. Female participants were significantly less likely to share their browser data, as were participants who were shown the browser data we asked to collect. Overall, we show the important role of user demographics in the ongoing work that intends to assess fingerprinting risks and improve user privacy, with findings to inform future privacy enhancing browser developments. The dataset and data collection tool we provide can be used to further study research questions not addressed in this work.

New regulations and increased awareness of data privacy have led to the deployment of new and more efficient differentially private mechanisms across both public institutions and industries. With the growing adoption of differential privacy, there is also a risk of introducing bugs into both the derivation of new mechanisms and their implementation. Ensuring these mechanisms is therefore crucial to ensure proper protection of data. However since differential privacy is not a property of a single output of a mechanism but a property of the mechanism itself, testing whether a mechanism is differentially private is not a trivial task. While ad hoc testing techniques exist under specific assumptions, no concerted effort has been made by the research community to develop a flexible and extendable tool for testing differentially private mechanisms. This paper introduces DP-Auditorium as a step advancing research in this direction. The main idea behind DP-Auditorium is to abstract the problem of testing differential privacy into two steps: (1) measuring the distance between distributions, and (2) finding neighboring datasets where a mechanism generates output distributions maximizing such distance. From a technical point of view, we propose three new algorithms for evaluating the distance between distributions. While these algorithms are well-known in the statistics community, we provide new estimation guarantees by leveraging the fact that we are only interested in verifying whether a mechanism is differentially private, and not on obtaining an exact estimate of the distance between two distributions. DP-Auditorium is easily extensible, as demonstrated in this paper by implementing a well-known approximate differential privacy testing algorithm to our library. Finally, we provide an extensive comparison to date of multiple testers across varying sample sizes and differential privacy parameters, demonstrating that there is no single tester that dominates all others, and that in order to ensure proper testing of mechanisms, one requires a combination of different techniques.

A lexicographic maximum of a set X subset of R-n is a vector in X whose smallest component is as large as possible, and subject to that requirement, whose second smallest component is as large as possible, and so on for the third smallest component, etc. Lexicographic maximization has numerous practical and theoretical applications, including fair resource allocation, analyzing the implicit regularization of learning algorithms, and characterizing refinements of game-theoretic equilibria. We prove that a minimizer in X of the exponential loss function L-c(x) = Sigma(i) exp(-cx(i)) converges to a lexicographic maximum of X as c -> infinity, provided that X is stable in the sense that a well-known iterative method for finding a lexicographic maximum of X cannot be made to fail simply by reducing the required quality of each iterate by an arbitrarily tiny degree. Our result holds for both near and exact minimizers of the exponential loss, while earlier convergence results made much stronger assumptions about the set X and only held for the exact minimizer. We are aware of no previous results showing a connection between the iterative method for computing a lexicographic maximum and exponential loss minimization. We show that every convex polytope is stable, but that there exist compact, convex sets that are not stable. We also provide the first analysis of the convergence rate of an exponential loss minimizer (near or exact) and discover a curious dichotomy: While the two smallest components of the vector converge to the lexicographically maximum values very quickly (at roughly the rate log n/c), all other components can converge arbitrarily slowly.

We study differentially private mechanisms for sharing training data in machine learning settings. Our goal is to enable learning of an accurate predictive model while protecting the privacy of each user's label. Previous work established privacy guarantees that assumed the features are public and given exogenously, a setting known as label differential privacy. In some scenarios, this can be a strong assumption that removes the interplay between features and labels from the privacy analysis. We relax this approach and instead assume the features are drawn from a distribution that depends on the private labels. We first show that simply adding noise to the label, as in previous work, can lead to an arbitrarily weak privacy guarantee, and also present methods for estimating this privacy loss from data. We then present a new mechanism that replaces some training examples with synthetically generated data, and show that our mechanism has a much better privacy-utility tradeoff if the synthetic data is realistic, in a certain quantifiable sense. Finally, we empirically validate our theoretical analysis.

We present new mechanisms for label differential privacy, a relaxation of differentially private machine learning that only protects the privacy of the labels in the training set. Our mechanisms cluster the examples in the training set using their (non-private) feature vectors, randomly re-sample each label from examples in the same cluster, and output a training set with noisy labels as well as a modified version of the true loss function. We prove that when the clusters are both large and high-quality, the model that minimizes the modified loss on the noisy training set converges to small excess risk at a rate that is comparable to the rate for non-private learning. We describe both a centralized mechanism in which the entire training set is stored by a trusted curator, and a distributed mechanism where each user stores a single labeled example and replaces her label with the label of a randomly selected user from the same cluster. We also describe a learning problem in which large clusters are necessary to achieve both strong privacy and either good precision or good recall. Our experiments show that randomizing the labels within each cluster significantly improves the privacy vs. accuracy trade-off compared to applying uniform randomized response to the labels, and also compared to learning a model via DP-SGD.

Data anonymization is an approach to privacypreserving data release aimed at preventing participants reidentification, and it is an important alternative to differential privacy in applications that cannot tolerate noisy data. Existing algorithms for enforcing k-anonymity in the released data assume that the curator performing the anonymization has complete access to the original data. Reasons for limiting this access range from undesirability to complete infeasibility. This paper explores ideas — objectives, metrics, protocols, and extensions — for reducing the trust that must be placed in the curator, while still maintaining a statistical notion of k-anonymity. We suggest trust (amount of information provided to the curator) and privacy (anonymity of the participants) as the primary objectives of such a framework. We describe a class of protocols aimed at achieving these goals, proposing new metrics of privacy in the process, and proving related bounds. We conclude by discussing a natural extension of this work that completely removes the need for a central curator. 1. Releasing private data (Background) As the use of big data continues to permeate modern society, so does the sharing of our personal data with centralized third-parties. For example, the U.S. Census Bureau shares aggregated population statistics with lawmakers (Abowd, 2018), and hospitals share medical information with insurance companies (Crellin & BCE, 2011). If unregulated, this type of information poses a threat to individual privacy. A trivial way to completely protect the privacy of individuals would be to simply not share any of their information, but such an absolutist approach is neither feasible nor useful. Google Research, New York, US Gatsby Unit, University College London, UK. Correspondence to: Gecia Bravo-Hermsdorff . A sensible compromise is to develop methods that balance the usefulness of the data against the privacy lost by the individuals. Two common frameworks for privacy-preserving data release are: differential privacy, i.e., DP (and its various extensions, e.g., Rényi differential privacy) and k-anonymity (and its various extensions, e.g., t-closeness). 1.1. A quick (incomplete) summary of DP In the central model of differential privacy (Dwork et al., 2006), a trusted curator stores the database, and an analyst1 issues queries about the database to a curator, who returns noisy responses. Such an approach requires the users to trust the curator with the entirety of their private data. Several models have been proposed to relax this requirement. In the local model, each user adds noise to their own data and responds to the analyst directly (Evfimievski et al., 2003). In the shuffle model, each user encrypts their noisy data (such that only the analyst may read them), and sends them to a trusted shuffler. The shuffler then randomly permutes these encrypted messages before forwarding them to the analyst (Cheu et al., 2019). 1.2. A quick (incomplete) summary of k-anonymity A dataset satisfies k-anonymity if for every individual whose data is contained in the dataset, their data are indistinguishable from that of at least k − 1 other individuals (also presented in this dataset). Since k-anonymity was first introduced (Sweeney, 2002), efficient algorithms for anonymizing a database (while preserving the maximum amount of information possible) have received increasing interest. Local suppression algorithms aim to achieve this by redacting specific (feature, user) entries of the database (Meyerson & Williams, 2004), while global suppression algorithms redact the same set of features for every user (El Emam et al., 2009). Note that here the “analyst” and the “public” are the same entity since the data observed by the analyst could be seen by anyone else. ar X iv :2 20 1. 12 30 6v 1 [ cs .D S] 2 8 Ja n 20 22 Statistical anonymity (Meyerson & Williams, 2004) showed that the problem of optimally anonymizing a database by either local and global suppression is NP-hard. In light of these results, several approximation algorithms have been proposed, particularly for local suppression (Aggarwal et al., 2005; Gkoulalas-Divanis et al., 2014). Similar to the central model of differential privacy, these algorithms/curators require access to the entire private data. Unlike differential privacy, variants of k-anonymity that reduce the trust that participants must place in the curator remain relatively unexplored. 2. Why we focus on k-anonymity (Motivation) Differential Privacy (DP) (Dwork et al., 2014) is a measure of privacy loss (typically denoted by ε) that holds true no matter what (e.g., even if additional information is released in the future). As a result of this strong propriety, any DP algorithm must be stochastic (e.g., by adding noise to the data). This, however, can be undesirable in a variety of applications (see Section 2.3 for examples). In contrast, while k-anonymity can be satisfied without adding noise to the data, its the privacy guarantee are contingent on the availability of auxiliary information (see (Narayanan & Shmatikov, 2008) for a famous example involving Netflix). 2.1. There is no panacea for private data As differential privacy offers an upper bound on each instance of privacy loss that holds regardless of anything else, it has a simple composition rule that can be invoked without further assumptions. Perhaps for this reason, DP is currently the de facto academic definition of privacy. As the issues surrounding privacy become increasingly pressing societal issues, it seems natural that the entities managing our private data would like to offer meaningful privacy guarantees. Unfortunately, despite being the “gold standard”, DP is often touted with essentially meaningless parameters (Domingo-Ferrer et al., 2021). For example, the US census of 2020 claims a “mathematical algorithm to ensure that the privacy of individuals is sufficiently protected” with a “budget” of ε = 19.61 (US Census Press Release CB21-CN.42). Setting aside a conspicuous similarity with the natural logarithm of the US population,2 the guarantee being made is essentially meaningless: “your participation in the census will not change the likelihood of any outcome by more than a factor of 331 million.” Given this clear rift in communication between theory and practice, it is fruitful to also consider privacy notions that The US population in 2020 is estimated at 331 million, and ln ( (331± 1) · 10 ) ≈ 19.617± 0.002. might have fewer “translation” issues, despite their technically “weaker” guarantees. 2.2. Natural extensions of k-anonymity For simplicity, consider the following setting: A database is to be released containing i.i.d. samples from the population, and the values can be split into two disjoint sets “Quasi-Identifiers” (QI) and “Sensitive Attributes” (SA). QI are not known to an adversary a priori, but could be learned (for some cost) via exogenous means. SA are features that are not known to the adversary, cannot be learned exogenously, and would be detrimental(valuable) to the participant(adversary) if learned by the adversary. Many “scalar-word” anonymity measures can be classified by the assumptions they make on the sensitive attributes (SA). The use of k-anonymity assumes that all SA are completely incomparable, while l-diversity (Machanavajjhala et al., 2007) allows for the possibility of identical SA (but is still blind to the magnitude of differences). Other metrics, such as t-closeness, δ-disclosure, and β-likeness, allow for a more general similarity metric between different SA (Khan et al., 2021). The main goal of this paper is to understand the trade-off between anonymization guarantees to the participants and the trust they must place in the entity performing the anonymization. We believe that k-anonymity is a suitable notion to use as a proof-of-concept to introduce such statistical relaxation. Extending this framework to more nuanced measures of anonymity would be of considerable practical interest. 2.3. Application examples Essentially, we consider a setting in which the private variables (the Sensitive Attributes) are incomparable (i.e., there is no metric of similarity) and unique (no two private variables are identical). In such a setting, k-anonymity is equivalent to l-diversity, and extensions such as t-closeness and β-likeness do not make sense (as the SA have no notion of similarity). For example, consider a database containing X-rays images (SA), along with some (Quasi-Identifying) demographics of the patients. The latter could likely be obtained by an adversary with minimal effort, whereas the former is essentially impossible to directly measure (without explicit cooperation from the individual). Given the exposing nature of these SA, it is not a stretch to think about an adversary using them for their personal gain at the expense of the owner of the images. Moreover, the details of everyone’s insides are rather unique. Another application is that of preventing browser fingerprinting (Laperdrix et al., 2020). Malicious websites engaged in browser fingerprinting query detailed information about a Statistical anonymity user’s device (e.g., which fonts they have installed). If these details are sufficiently unique, they can be used to covertly track a user across different the web. While certain system details can be made less amenable to fingerprinting by adding noise to them (e.g., window size/resolution), the option of returning noisy responses is often not practical (e.g., uncertain browser type). Several browsers have proposed to prevent fingerprinting by ensuring that the information queried by a website is always k-anonymous, and blocking the query otherwise. However, the only way to completely guarantee k-anonymity is to grant a central curator access to the full data of every user. 3. The big picture (What we did) In seeking a version of k-anonymity that does not require a fully-trusted cu

Modern statistical estimation is often performed in a distributed setting where each sample belongs to a single user who shares their data with a central server. Users are typically concerned with preserving the privacy of their samples, and also with minimizing the amount of data they must transmit to the server. We give improved private and communication-efficient algorithms for estimating several popular measures of the entropy of a distribution. All of our algorithms have constant communication cost and satisfy local differential privacy. For a joint distribution over many variables whose conditional independence is given by a tree, we describe algorithms for estimating Shannon entropy that require a number of samples that is linear in the number of variables, compared to the quadratic sample complexity of prior work. We also describe an algorithm for estimating Gini entropy whose sample complexity has no dependence on the support size of the distribution and can be implemented using a single round of concurrent communication between the users and the server. In contrast, the previously best-known algorithm has high communication cost and requires the server to facilitate interaction between the users. Finally, we describe an algorithm for estimating collision entropy that generalizes the best known algorithm to the private and communication-efficient setting.

We study the problem of designing privacy-enhanced solutions for interest-based advertisement (IBA). IBA is a key component of the online ads ecosystem and provides a better ad experience to users. Indeed, IBA enables advertisers to show users impressions that are relevant to them. Nevertheless, the current way ad tech companies achieve this is by building detailed interest profiles for individual users. In this work we ask whether such fine grained personalization is required, and present mechanisms that achieve competitive performance while giving privacy guarantees to the end users. More precisely we present the first detailed exploration of how to implement Chrome's Federated Learning of Cohorts (FLoC) API. We define the privacy properties required for the API and evaluate multiple hashing and clustering algorithms discussing the trade-offs between utility, privacy, and ease of implementation.

We study the problem of differentially private optimization with linear constraints when the right-hand-side of the constraints depends on private data. This type of problem appears in many applications, especially resource allocation. Previous research provided solutions that retained privacy but sometimes violated the constraints. In many settings, however, the constraints cannot be violated under any circumstances. To address this hard requirement, we present an algorithm that releases a nearly-optimal solution satisfying the constraints with probability 1. We also prove a lower bound demonstrating that the difference between the objective value of our algorithm's solution and the optimal solution is tight up to logarithmic factors among all differentially private algorithms. We conclude with experiments demonstrating that our algorithm can achieve nearly optimal performance while preserving privacy.

We study the cost sharing problem for cooperative games in situations where the cost function C is not available via oracle queries, but must instead be learned from samples drawn from a distribution, represented as tuples (S, C(S)), for different subsets S of players. We formalize this approach, which we call statistical cost sharing, and consider the computation of the core and the Shapley value. Expanding on the work by Balcan et al, we give precise sample complexity bounds for computing cost shares that satisfy the core property with high probability for any function with a non-empty core. For the Shapley value, which has never been studied in this setting, we show that for submodular cost functions with curvature bounded curvature kappa it can be approximated from samples from the uniform distribution to a sqrt{1 - kappa} factor, and that the bound is tight. We then define statistical analogues of the Shapley axioms, and derive a notion of statistical Shapley value and that these can be approximated arbitrarily well from samples from any distribution and for any function.

Many enterprises have migrated their data from an on-site database to a cloud-based database-as-a-service that handles all database-related administrative tasks while providing a simple SQL interface to the end user. Businesses are also increasingly relying on machine learning to understand their customers and develop new products. Given these converging trends, there is a pressing need for database-as-a-service providers to add support for sophisticated machine learning algorithms to the core functionality of their products.

As the number of ad exchanges has grown, publishers have turned to low regret learning algorithms to decide which exchange offers the best price for their inventory. This in turn opens the following question for the exchange: how to set prices to attract as many sellers as possible and maximize revenue. In this work we formulate this precisely as a learning problem, and present algorithms showing that by simply knowing that the counterparty is using a low regret algorithm is enough for the exchange to have its own low regret learning algorithm to find the optimal price.

Ad exchange platforms connect online publishers and advertisers and facilitate the sale of billions of impressions every day. We study these environments from the perspective of a publisher who wants to find the profit-maximizing exchange in which to sell his inventory. Ideally, the publisher would run an auction among exchanges. However, this is not usually possible due to practical business considerations. Instead, the publisher must send each impression to only one of the exchanges, along with an asking price. We model the problem as a variation of the multi-armed bandits problem in which exchanges (arms) can behave strategically in order to maximizes their own profit. We propose e mechanisms that find the best exchange with sub-linear regret and have desirable incentive properties.