Public sector AI procurement checklists can help guide efforts to create regulatable AI systems.

There is increasing attention being given to how to regulate AI systems. As governing bodies grapple with what values to encapsulate into regulation, we consider the technical half of the question: To what extent can AI experts vet an AI system for adherence to regulatory requirements? We investigate this question through the lens of two public sector procurement checklists, identifying what we can do now, what should be possible with technical innovation, and what requirements need a more interdisciplinary approach.

Statistical measures for group fairness in machine learning reflect the gap in performance of algorithms across different groups. These measures, however, exhibit a high variance between different training instances, which makes them unreliable for empirical evaluation of fairness. What causes this high variance? We investigate the impact on group fairness of different sources of randomness in training neural networks. We show that the variance in group fairness measures is rooted in the high volatility of the learning process on under-represented groups. Further, we recognize the dominant source of randomness as the stochasticity of data order during training. Based on these findings, we show how one can control group-level accuracy (i.e., model fairness), with high efficiency and negligible impact on the model's overall performance, by simply changing the data order for a single epoch.

The privacy risks of machine learning models is a major concern when training them on sensitive and personal data. We discuss the tradeoffs between data privacy and the remaining goals of trustworthy machine learning (notably, fairness, robustness, and explainability).

Complex black-box machine learning models are regularly used in critical decision-making domains. This has given rise to several calls for algorithmic explainability. Many explanation algorithms proposed in literature assign importance to each feature individually. However, such explanations fail to capture the joint effects of sets of features. Indeed, few works so far formally analyze high-dimensional model explanations. In this paper, we propose a novel high dimension model explanation method that captures the joint effect of feature subsets. We propose a new axiomatization for a generalization of the Banzhaf index; our method can also be thought of as an approximation of a black-box model by a higher-order polynomial. In other words, this work justifies the use of the generalized Banzhaf index as a model explanation by showing that it uniquely satisfies a set of natural desiderata and that it is the optimal local approximation of a black-box model. Our empirical evaluation of our measure highlights how it manages to capture desirable behavior, whereas other measures that do not satisfy our axioms behave in an unpredictable manner.

Privacy and transparency are two key foundations of trustworthy machine learning. Model explanations offer insights into a model's decisions on input data, whereas privacy is primarily concerned with protecting information about the training data. We analyze connections between model explanations and the leakage of sensitive information about the model's training set. We investigate the privacy risks of feature-based model explanations using membership inference attacks: quantifying how much model predictions plus their explanations leak information about the presence of a datapoint in the training set of a model. We extensively evaluate membership inference attacks based on feature-based model explanations, over a variety of datasets. We show that backpropagation-based explanations can leak a significant amount of information about individual training datapoints. This is because they reveal statistical information about the decision boundaries of the model about an input, which can reveal its membership. We also empirically investigate the trade-off between privacy and explanation quality, by studying the perturbation-based model explanations.

An important and surprising phenomenon in voting theory is the No-Show Paradox (NSP), which occurs if a voter is better off by abstaining from an election. While it is known that certain voting rules suffer from this paradox in principle, the extent to which it is of practical concern is not well understood. We aim at filling this gap by analyzing the likelihood of the NSP for six Condorcet extensions (Black’s rule, Baldwin’s rule, Nanson’s rule, Max-Min, Tideman’s rule, and Copeland’s rule) under various preference models using Ehrhart theory as well as extensive computer simulations. We find that, for few alternatives, the probability of the NSP is rather small (less than 4% for four alternatives and all considered preference models, except for Copeland’s rule). As the number of alternatives increases, the NSP becomes much more likely and which rule is most susceptible to abstention strongly depends on the underlying distribution of preferences.

Can an adversary exploit model explanations to infer information about the training set? To investigate this question, we focus on membership inference attacks: given a datapoint and a model explanation, the attacker’s goal is to decide whether the point belongs to the training data. We study this problem for two popular transparency methods: gradient-based attribution methods and record-based influence measures. We develop membership inference attacks based on these model explanations, and extensively test them on a variety of datasets. For gradient-based methods, we develop an attack that can be executed by an attacker that has very limited resources, while maintaining comparable accuracy to existing membership inference attacks. We show that record-based measures can be effectively utilized for membership inference attacks; moreover, we demonstrate that they can be exploited to recover significant parts of the training set. Finally, our results indicate that minorities and outliers are more vulnerable to these type of attacks than majority groups.

The fact that machine learning is growing more and more entrenched in almost every aspect of society, combined with the opacity of various of its algorithms has induced the relatively young research area of transparent machine learning. The aim of this domain is to provide explanations for automated decisions to increase public trust. In my thesis, I am going to consider certain problems that arise from this research agenda. Particularly, I so far considered, the dilemma of conflicting explanations and the issue of privacy concerns arising from transparency.

Can an adversary exploit model explanations to infer sensitive information about the models’ training set? To investigate this question, we €rst focus on membership inference aˆacks: given a data point and a model explanation, the aŠacker’s goal is to decide whether or not the point belongs to the training data. We study this problem for two popular transparency methods: gradient-based aŠribution methods and record-based inƒuence measures. We develop membership inference aŠacks based on these model explanations, and extensively test them on a variety of datasets. For gradient-based methods, we show that the explanations can leak a signi€cant amount of information about the individual data points in the training set, much beyond what is leaked through the predicted labels. We also show that record-based measures can be e‚ectively, and even more signi€cantly, exploited for membership inference attacks. More importantly, we design reconstruction aˆacks against this class of model explanations. We demonstrate that they can be exploited to recover signi€cant parts of the training set. Finally, our results indicate that minorities and outliers are more vulnerable to these type of aŠacks than the rest of the population. Œus, there is a signi€cant disparity for the privacy risks of model explanations across di‚erent groups.

Can an adversary exploit model explanations to infer sensitive information about the models' training set? To investigate this question, we first focus on membership inference attacks: given a data point and a model explanation, the attacker's goal is to decide whether or not the point belongs to the training data. We study this problem for two popular transparency methods: gradient-based attribution methods and record-based influence measures. We develop membership inference attacks based on these model explanations, and extensively test them on a variety of datasets. For gradient-based methods, we show that the explanations can leak a significant amount of information about the individual data points in the training set, much beyond what is leaked through the predicted labels. We also show that record-based measures can be effectively, and even more significantly, exploited for membership inference attacks. More importantly, we design reconstruction attacks against this class of model explanations. We demonstrate that they can be exploited to recover significant parts of the training set. Finally, our results indicate that minorities and outliers are more vulnerable to these type of attacks than the rest of the population. Thus, there is a significant disparity for the privacy risks of model explanations across different groups.

Results from voting theory are increasingly used when dealing with collective decision making in computational multiagent systems. An important and surprising phenomenon in voting theory is the No-Show Paradox (NSP), which occurs if a voter is better off by abstaining from an election. While it is known that certain voting rules suffer from this paradox in principle, the extent to which it is of practical concern is notwell understood. We aim at filling this gap by analyzing the likelihood of the NSP for six Condorcet extensions (Black's rule, Baldwin's rule, Nanson's rule, MaxiMin, Tideman's rule, and Copeland's rule) under various preference models using Ehrhart theory as well as extensive computer simulations. We find that, for few alternatives, the probability of the NSP is rather small (less than 4% for four alternatives and all considered preference models, except for Copeland's rule). As the number of alternatives increases, the NSP becomes much more likely and which rule is most susceptible to abstention strongly depends on the underlying distribution of preferences.

We study the following problem: given a labeled dataset and a specific datapoint x, how did the i-th feature influence the classification for x? We identify a family of numerical influence measures - functions that, given a datapoint x, assign a numeric value phi_i(x) to every feature i, corresponding to how altering i's value would influence the outcome for x. This family, which we term monotone influence measures (MIM), is uniquely derived from a set of desirable properties, or axioms. The MIM family constitutes a provably sound methodology for measuring feature influence in classification domains; the values generated by MIM are based on the dataset alone, and do not make any queries to the classifier. While this requirement naturally limits the scope of our framework, we demonstrate its effectiveness on data.

Recent years have seen the widespread implementation of data-driven algorithms making decisions in increasingly highstakes domains, such as finance, healthcare, transportation and public safety. Using novel ML techniques, these algorithms are able to process massive amounts of data and make highly accurate predictions; however, their inherent complexity makes it increasingly difficult for humans to understand why certain decisions were made. Indeed, these algorithms are black-box decision makers: their underlying decision processes are either hidden from human scrutiny by proprietary law, or (as is often the case) their inner workings are so complicated that even their own designers will be hard-pressed to explain the underlying reasoning behind their decision making processes. By obfuscating their function, data-driven classifiers run the risk of exposing human stakeholders to risks. These may include incorrect decisions (e.g. a loan application that was wrongly rejected due to system error), information leaks (e.g. an algorithm inadvertently uses information it should not have used), or discrimination (e.g. biased decisions against certain ethnic or gender groups). Government bodies and regulatory authorities have recently begun calling for algorithmic transparency: providing human-interpretable explanations of the underlying reasoning behind large-scale decision making algorithms. My thesis research will be concerned with an axiomatic analysis of automatically generated explanations of such classifiers. Especially, I'm interested in how to decide which explanation of a decision to trust given that there are many, potentially conflicting, possible explanations for any given decision.

Pirate syndicates capturing tankers to siphon oil, causing an estimated cost of $5 billion a year, has become a serious security issue for maritime traffic. in response to the threat, coast guards and navies deploy patrol boats to protect international oil trade. However, given the vast area of the sea and the highly time and space dependent behaviors of both players, it remains a significant challenge to find efficient ways to deploy patrol resources. In this paper, we address the research challenges and provide four key contributions. First, we construct a Stackelberg model of the oil-siphoning problem based on incident reports of actual attacks; Second, we propose a compact formulation and a constraint generation algorithm, which tackle the exponentially growth of the defender's and attacker's strategy spaces, respectively, to compute efficient strategies of security agencies; Third, to further improve the scalability, we propose an abstraction method, which exploits the intrinsic similarity of defender's strategy space, to solve extremely large-scale games; Finally, we evaluate our approaches through extensive simulations and a detailed case study with real ship traffic data. The results demonstrate that our approach achieves a dramatic improvement of scalability with modest influence on the solution quality and can scale up to realistic-sized problems.

In this work, we focus on local explanations for data analytics; in other words: given a datapoint (cid:126) x , how important was the i -th feature in determining the outcome for (cid:126) x ? The literature has seen a recent emergence of various analytical answers to this question. We argue for a linear influence measure explanation: given a datapoint (cid:126) x , assign a value φ i ( (cid:126) x ) to every feature i , which roughly corresponds to feature i ’s importance in determining the outcome for (cid:126) x . We present a family of measures called MIM (monotone influence measures), that are uniquely derived from a set of axioms: desirable properties that any reasonable influence measure should satisfy. Departing from prior work on influence measures, we assume no knowledge — or access — to the underlying classifier labeling the dataset. In other words, our influence measures are based on the dataset alone and do not make any queries to the classi-fier. We compare MIM to other linear explanation models in the literature and discuss their underlying assumptions, merits, and limitations.

In this work we focus on the following question: how important was the i-th feature in determining the outcome for a given datapoint? We identify a family of influence measures; functions that, given a datapoint x, assign a value phi_i(x) to every feature i, which roughly corresponds to that i's importance in determining the outcome for x. This family is uniquely derived from a set of axioms: desirable properties that any reasonable influence measure should satisfy. Departing from prior work on influence measures, we assume no knowledge of - or access to - the underlying classifier labelling the dataset. In other words, our influence measures are based on the dataset alone, and do not make any queries to the classifier. While this requirement naturally limits the scope of explanations we provide, we show that it is effective on real datasets.