This is an erratum for the article “Annotated Automatic Pruning of Universal Probabilistic Programming Languages” published in ACM Trans. Probab. Mach. Learn. 1, 3, Article 15 (August 2025), 33 pages.
We address efficient Bayesian inference in multilevel models, where group-specific latent variables are drawn from a shared hyperprior. In standard approaches, inferring the posterior for a new group requires revisiting all previous groups, incurring growing computational cost due to increased data volume and latent dimensionality. We propose replacing past groups with a set of weighted virtual observations of latent variables that preserve the prior over new groups, enabling fast, scalable inference. We provide theoretical analysis, empirical validation on case studies, and a reference implementation compatible with common probabilistic programming languages and inference algorithms.
Deep ensembles (DE) are widely recognized for their robustness and improved uncertainty quantification in machine learning tasks. However, a persistent challenge lies in ensuring sufficient diversity among ensemble members: independently trained models often converge to similar solutions, limiting the ensemble’s overall effectiveness. In this work, we propose a novel Wasserstein regularization approach in function space. Specifically, we maximize pairwise Wasserstein or heuristic Gromov-Wasserstein (GW) distances between hidden layers’ representations of ensemble members. By interpreting these representations as empirical probability distributions, our method leverages the geometry of optimal transport to enforce functional diversity. Furthermore, we demonstrate that regularizing in function space—via distributional alignment using Wasserstein-based metrics—yields better diversity and generalization than approaches based on weight space alone. Empirical results on benchmark datasets demonstrate that our method outperforms standard DE and weight-space regularized variants in both predictive accuracy and uncertainty estimation. Code and experiments are available at: https://github.com/leotaku98/wass_de .
In the context of probabilistic programming languages, we put forward an approach to formal semantics and sampling-based inference with guarantees, centred on an action-based language equipped with a small-step operational semantics. We argue that this choice offers benefits in terms of clarity and effective, vectorised implementations. In measure-theoretic terms, a product of Markov kernels is used to formalise the small-step operational semantics. A trace semantics is also introduced based on a probability space of infinite sequences, along with a finite approximation theorem, relating the exact semantics to a truncated-execution semantics. This result directly leads to a sampling algorithm with guarantees that can be efficiently SIMD-parallelized. Experiments conducted with an implementation based on TensorFlow show that our approach compares very favourably to state-of-the-art tools for probabilistic programming and inference.
Automatic software generation based on some specification is known as program synthesis . Most existing approaches formulate program synthesis as a search problem with discrete parameters. In this article, we present a novel formulation of program synthesis as a continuous optimization problem using an evolutionary approach, known as Covariance Matrix Adaptation Evolution Strategy. We then propose several mapping schemes to convert the continuous formulation into actual programs and propose different restart policies for the evolutionary approach. This is the first work that demonstrates the feasibility of continuous approach in synthesizing complex programs, not just simple toy programs. We compare our system, Genesys , to several recent program synthesis techniques (in both discrete and continuous domains). We find that Genesys synthesizes more programs within a fixed time budget than those existing schemes. For example, for programs of length 10, Genesys synthesizes 28% more programs than those existing schemes within the same time budget.
This paper analyzes neural networks through graph variables and statistical sufficiency. We interpret neural network layers as graph-based transformations, where neurons act as pairwise functions between inputs and learned anchor points. Within this formulation, we establish conditions under which layer outputs are sufficient for the layer inputs, that is, each layer preserves the conditional distribution of the target variable given the input variable. Under dense anchor point assumptions, we prove that asymptotic sufficiency holds in the infinite-width limit and is preserved throughout training. To align more closely with practical architectures, we further show that sufficiency can be achieved with finite-width networks by assuming region-separated input distributions and constructing appropriate anchor points. Our framework covers fully connected layers, general pairwise functions, ReLU and sigmoid activations, and convolutional neural networks. This work bridges statistical sufficiency, graph-theoretic representations, and deep learning, providing a new statistical understanding of neural networks.
We present a framework for adaptive-stepsize MCMC sampling based on time-rescaled Langevin dynamics, in which the stepsize variation is dynamically driven by an additional degree of freedom. Our approach augments the phase space by an additional variable which in turn defines a time reparameterization. The use of an auxiliary relaxation equation allows accumulation of a moving average of a local monitor function and provides for precise control of the timestep while circumventing the need to modify the drift term in the physical system. Our algorithm is straightforward to implement and can be readily combined with any off-the-peg fixed-stepsize Langevin integrator. As a particular example, we consider control of the stepsize by monitoring the norm of the log-posterior gradient, which takes inspiration from the Adam optimizer, the stepsize being automatically reduced in regions of steep change of the log posterior and increased on plateaus, improving numerical stability and convergence speed. As in Adam, the stepsize variation depends on the recent history of the gradient norm, which enhances stability and improves accuracy compared to more immediate control approaches. We demonstrate the potential benefit of this method–both in accuracy and in stability–in numerical experiments including Neal's funnel and a Bayesian neural network for classification of MNIST data.
We propose a novel algorithm for combined unit and layer pruning of deep neural networks that functions during training and without requiring a pre-trained network to apply. Our algorithm optimally trades-off learning accuracy and pruning levels while balancing layer vs. unit pruning and computational vs. parameter complexity using only three user-defined parameters, which are easy to interpret and tune. We formulate a stochastic optimization problem over the network weights and the parameters of variational Bernoulli distributions for binary Random Variables taking values either 0 or 1 and scaling the units and layers of the network. Optimal network structures are found as the solution to this optimization problem. Pruning occurs when a variational parameter converges to 0 rendering the corresponding structure permanently inactive, thus saving computations both during training and prediction. A key contribution of our approach is to define a cost function that combines the objectives of prediction accuracy and network pruning in a computational/parameter complexity-aware manner and the automatic selection of the many regularization parameters. We show that the proposed algorithm converges to solutions of the optimization problem corresponding to deterministic networks. We analyze the ODE system that underlies our stochastic optimization algorithm and establish domains of attraction for the dynamics of the network parameters. These theoretical results lead to practical pruning conditions avoiding the premature pruning of units and layers during training. We evaluate our method on the CIFAR-10/100 and ImageNet datasets using ResNet architectures and demonstrate that it gives improved results with respect to pruning ratios and test accuracy over layer-only or unit-only pruning and favorably competes with combined unit and layer pruning algorithms requiring pre-trained networks.
Automatic software generation based on some specification is known as program synthesis. Most existing approaches formulate program synthesis as a search problem with discrete parameters. In this paper, we present a novel formulation of program synthesis as a continuous optimization problem using an evolutionary approach, known as Covariance Matrix Adaptation Evolution Strategy. We then propose several mapping schemes to convert the continuous formulation into actual programs and propose different restart policies for the evolutionary approach. This is the first work that demonstrates the feasibility of continuous approach in synthesizing complex programs, not just simple toy programs. We compare our system, Genesys , to several recent program synthesis techniques (in both discrete and continuous domains). We find that Genesys synthesizes more programs within a fixed time budget than those existing schemes. For example, for programs of length 10, Genesys synthesizes 28% more programs than those existing schemes within the same time budget.
Machine Learning (ML) is widely used in various domains but is susceptible to biases that can lead to unfair decisions. Bias can arise from biased data, algorithms, or data collection processes, making it crucial to develop methods that ensure fairness. This article introduces the Detect and Alleviate Bias (DAB) framework, a novel approach designed to identify and mitigate bias in ML models, focusing on sensitive attributes such as gender and race. The key contributions of DAB include: (1) a holistic pipeline that combines data pre-processing, model enhancement, situation testing, and bias mitigation techniques; (2) the application of Counterfactual Fairness testing to detect individual biases; and (3) the integration of multiple bias mitigation strategies to improve fairness in binary classification tasks. We demonstrate the practical implications of DAB through empirical experiments on two widely used datasets, showing that it reduces bias and improves fairness with a slight compromise in model performance, with a significant potential for making an impact in sensitive domains such as healthcare and criminal justice. The results show that pre-processing and post-processing mitigation techniques achieve the best improvements in fairness for unrepresentative datasets, while in-processing methods are more effective in enhancing fairness for representative datasets; however, an inverse relationship between model performance and fairness is observed.
Conditional independence between variables in Gaussian graphical models (also known as Gaussian Markov random fields) is represented by the conditional independence graph, \(G\). Most approaches for inferring conditional independence graph rely on the penalized log-likelihood, where a regularization hyperparameter, \(\lambda\), controls the preference for either a sparsely or densely connected solution. In this article, we present a method for selecting \(\lambda\) based on the minimum description length (MDL) principle. Our approach improves upon previous methods by better accounting for \(G\) using our novel graph coders. Experiments on known Gaussian graphical models demonstrate that our approach has a higher F1 score in recovering the true conditional independence graph than existing methods, especially when the number of observations is small compared to the number of variables. We also applied our method to a real-world electrocardiogram (ECG) dataset to investigate the inferred conditional independence graph in healthy subjects versus a group of subjects with Kawasaki disease. Finally, we used the learned conditional independence graphs for the classification of healthy subjects versus those with Kawasaki disease.
Leveraging prior experience has been effective in accelerating reinforcement learning (RL). One approach for leveraging prior experience consists of learning skills, which are temporally extended sequences of actions, and priors over those skills, which are then used to learn new tasks. Existing methods either learn single-length skills or learn skills with every possible length, e.g., options. Instead, we propose using a more select set of skill lengths, where the agent can learn repetitive, monotonous sequences of actions as “long” skills, and unique, highly complex sequences of actions as “short” skills. To implement that intuition, we introduce multi-length skills with priors (MLSP) for RL, a novel approach that leverages prior experience and presents a new RL formulation that considers multi-length skills. We, first, introduce hierarchical multi-length skill learning (HIMES), which is a deep latent variable model that learns multi-length skills by hierarchical composing skills to form longer skills, then we provide a notion of optimal length based on reconstruction error, from which a length prior is learned, and lastly a prior over skills is learned. Finally, MLSP integrates these models to guide downstream learning, where now two policies are learned: The length policy that selects which skill length to use, and then the skill policy that selects what skill of a particular skill length to use. We validate our approach on complex robotic manipulation tasks and show that MLSP significantly accelerates downstream task learning with respect to competitive benchmarks. Code and videos of the trained agents are available at https://github.com/cijerezg/MLSP.
In recent years, neural networks (NNs) have been embraced by several scientific and engineering disciplines for diverse modeling and inferencing applications. The importance of quantifying the confidence in NN predictions has escalated due to the increasing adoption of these decision models. Nevertheless, conventional NN do not furnish uncertainty estimates associated with their predictions and are therefore ill-calibrated. Uncertainty quantification techniques offer probability distributions or CIs to represent the uncertainty associated with NN predictions, instead of solely presenting the point predictions/estimates. Once the uncertainty in NN is quantified, it is crucial to leverage this information to modify training objectives and improve the accuracy and reliability of the corresponding decision models. This work presents a novel framework to utilize the knowledge of input and output uncertainties in NN to guide querying process in the context of Active Learning. We also derive the lower and upper bounds for label complexity. The efficacy of the proposed framework is established by conducting experiments across classification and regression tasks.
Synchronous languages are now a standard industry tool for critical embedded systems. Designers write high-level specifications by composing streams of values using block diagrams. These languages have been recently extended with Bayesian reasoning to program state-space models which compute a stream of distributions given a stream of observations. Yet, the semantics of probabilistic models is only defined for scheduled equations – a significant limitation compared to dataflow synchronous languages and block diagrams. In this paper we propose a new operational semantics and a new denotational semantics for a probabilistic synchronous language that are both schedule agnostic. The key idea is to externalize the source of randomness and interpret a probabilistic expression as a stream of functions mapping random elements to a value and positive score. The operational semantics interprets expressions as state machines where mutually recursive equations are evaluated using a fixpoint operator. The denotational semantics directly manipulates streams and is thus a better fit to reason about program equivalence. We use the denotational semantics to prove the correctness of a program transformation required to run an optimized inference algorithm for state-space models with constant parameters.
Probabilistic programming languages aim to separate the user-specified model from the inference runtime system, allowing users to focus on model design and leave inference to the compiler and runtime systems. However, the structure of the model can affect the inference efficiency, something that is often utilized when implementing Bayesian inference solutions without probabilistic programming. In this paper, we consider a specific class of important models within phylogenetics, which are used for tree inference. Algorithms utilizing the structures of trees in such models have been used for a long time within phylogenetics, outside the scope of probabilistic programming. In particular, the use of belief propagation can significantly speed up probabilistic tree inference when combined with other approximate inferences, such as MCMC. However, there is an open problem of how to incorporate both belief-propagation (marginalization) of a subset of latent variables in typed universal probabilistic programming languages and still be able to apply approximate inference for the rest of the variables. In this paper, we propose a solution to this problem by using an approach where a user only needs to mark which latent variables in the program should be marginalized, and the rest of the belief propagation is handled automatically. We call this approach annotated automatic pruning, acknowledging the original terminology of pruning from the phylogenetic literature. We evaluate our approach using non-trivial models for tree inference, assessing performance, and testing correctness by comparing to MrBayes as the reference implementation.
Probabilistic programming languages are used to write probabilistic models to make probabilistic inferences. A number of rigorous semantics have been developed to reason about transformations of probabilistic programs and their execution. We propose to investigate the formalization of such a semantics to allow for the formal verification of probabilistic programs. Concretely, we extend an existing formalization of measure and integration theory in the Rocq prover, a proof assistant based on dependent type theory, with s-finite kernels, a mathematical structure to interpret typing judgments in the semantics of a probabilistic programming language. We explain the issues raised by organizing kernels as a hierarchy of mathematical structures and use the latter to formalize the semantics of a first-order probabilistic programming language. We use this semantics to establish generic properties of this language, prove rewriting laws to perform symbolic evaluation, and reason about iteration.
Probabilistic programming languages (PPLs) are becoming increasingly important in many scientific disciplines, such as economics, epidemiology, and biology, to extract meaning from sources of data while accounting for one's uncertainty. The key idea of probabilistic programming is to decouple inference and model specification, thus allowing the practitioner to approach their task at hand using Bayesian inference, without requiring extensive knowledge in programming or computational statistics. At the same time, the complexity of problem settings in which PPLs are employed is steadily increasing, both in terms of project size and model complexity, calling for more flexible and efficient systems. In this work, we describe Turing.jl , a general-purpose PPL, which is designed to be flexible, efficient, and easy to use. Turing.jl is built on top of the Julia programming language, which is known for its high performance and ease-of-use. We describe the design of Turing.jl , contextualizing it within different types of users and use cases, its key features, and how it can be used to solve a wide range of problems. We also provide a brief overview of the ecosystem around Turing.jl , including the different libraries and tools that can be used in conjunction with it. Finally, we provide a few examples of how Turing.jl can be used in practice.
Quantitative characterizations and estimations of uncertainty are of fundamental importance for machine learning classification, particularly in safety-critical settings where continuous real-time monitoring requires explainable and reliable scoring. Reliance on the maximum a posteriori principle to determine label classification can obscure the certainty of a label assignment. We develop a theoretical framework for quantitative scores of certainty and competence based on predicted probability estimates, formally prove their properties, and empirically confirm the inferential power of these properties across different data modalities, tasks and model architectures. Our theoretical results establish that competent models have distinct distributions of certainty for true and false positives conditioned on inputs similar to training and testing data, and prove that this framework provides a reliable means to infer the quality of model predictions and detect false positives. Our empirical results bear out that there are distinct distributions of certainty scores on training and holdout data, as well as data that is a priori out-of-distribution. For expert models, at least 62.1% of false positives could be identified when using a cut-off at at the bottom 5% TP threshold. Further, we found a strong negative correlation between empirical competence and the FPR95TPR rate for EnergyBased out-of-distribution (OOD) detectors. Finally, we developed two forms of an OOD detector that were able to reliably distinguish in-distribution data from OOD data for both frequentist and Bayesian models, performing better on average than previous state-of-the-art EnergyBased OOD detection methods, and improving upon the baseline Monte Carlo Dropout AUPR-OUT performance on average by 14.4% and 16.5%, and reducing the FPR95TPR by 54.2% and 37.6%.
As data-driven and AI-based decision making gains widespread adoption across disciplines, it is crucial that both data privacy and decision fairness are appropriately addressed. Although differential privacy (DP) provides a robust framework for guaranteeing privacy and methods are available to improve fairness, most prior work treats the two concerns separately. Even though there are existing approaches that consider privacy and fairness simultaneously, they typically focus on a single specific learning task, limiting their generalizability. In response, we introduce SAFES, a Sequential PrivAcy and Fairness Enhancing data Synthesis procedure that sequentially combines DP data synthesis with a fairness-aware data preprocessing step. SAFES allows users flexibility in navigating the privacy-fairness-utility trade-offs. We illustrate SAFES with different DP synthesizers and fairness-aware data preprocessing methods and run extensive experiments on multiple real datasets to examine the privacy-fairness-utility trade-offs of synthetic data generated by SAFES. Empirical evaluations demonstrate that for reasonable privacy loss, SAFES-generated synthetic data can achieve significantly improved fairness metrics with relatively low utility loss.
