Bayesian Classification and Regression Trees (BCART) and Bayesian Additive Regression Trees (BART) are popular Bayesian regression models widely applicable in modern regression problems. Their popularity is intimately tied to the ability to flexibly model complex responses depending on high-dimensional inputs while simultaneously being able to quantify uncertainties. This ability to quantify uncertainties is key, as it allows researchers to perform appropriate inferential analyses in settings that have generally been too difficult to handle using the Bayesian approach. However, surprisingly little work has been done to evaluate the sensitivity of these modern regression models to violations of modeling assumptions. In particular, we will consider influential observations, which one reasonably would imagine to be common—or at least a concern—in the big-data setting. In this article, we consider both the problem of detecting influential observations and adjusting predictions to not be unduly affected by such potentially problematic data. We consider three detection diagnostics for Bayesian tree models, one an analogue of Cook’s distance and the others taking the form of a divergence measure and a conditional predictive density metric, and then propose an importance sampling algorithm to re-weight previously sampled posterior draws so as to remove the effects of influential data in a computationally efficient manner. Finally, our methods are demonstrated on real-world data where blind application of the models can lead to poor predictions and inference. Supplementary materials for this article are available online.

Online reviews provide rich information on customer satisfaction, displaying various numeric ratings as well as detailed explanations presented in written form. However, analyzing such data is challenging due to the unstructured nature of text. This article introduces a novel machine-learning method for identifying interpretable key drivers of star ratings from text reviews, which might vary across segments. By adopting the Ising model prior to account for dependence between words, the model simultaneously achieves segmentation, identifies segment-level key topics (i.e., groups of frequently co-occurring words), and estimates the impacts of the selected words on the ratings. The authors first demonstrate that the proposed model successfully identifies segment-specific key drivers of customer satisfaction using illustrative simulated review data. Then, the authors utilize real-world reviews from Yelp for empirical applications. When applied to online reviews of 5,241 Arizona-based restaurants, the model identifies three distinct restaurant segments, each characterized by three to five important topics. The model's performance is evaluated against six benchmark models, encompassing various topic models and latent class regression with variable selection. The comparison results emphasize the proposed model's unique advantages in prediction, interpretability, and handling heterogeneity. Additionally, the authors demonstrate the applicability of the model in examining customer segmentation for individual restaurants.

Bayesian Additive Regression Trees (BART) has gained widespread popularity, inspiring numerous extensions across diverse applications. However, relatively little attention has been given to modeling dependent data. To fill this gap, we introduce Correlated BART (CBART), which extends BART to account for correlated errors. With a dummy representation, efficient matrix computation was developed for the estimation of CBART. Building on CBART, we propose CBART0x2010GP, a nonparametric regression model that integrates CBART with a Gaussian process (GP) in an additive framework. In CBART0x2010GP, CBART retrieves the true signal of covariates0x2010response relationship, while the GP extracts the dependency structure of residuals. To enable scalable inference of CBART0x2010GP, we develop a two0x2010stage analysis of variance with weighted residuals approach to substantially reduce the computational complexity. Simulation studies demonstrate that CBART-GP not only accurately recovers the true covariate0x2010response relationship but also achieves strong predictive performance. A real world application further illustrates its practical utility.

Many popular survival models rely on restrictive parametric, or semiparametric, assumptions that could provide erroneous predictions when the effects of covariates are complex. Modern advances in computational hardware have led to an increasing interest in flexible Bayesian nonparametric methods for time-to-event data such as Bayesian additive regression trees (BART). We propose a novel approach that we call nonparametric failure time (NFT) BART in order to increase the flexibility beyond accelerated failure time (AFT) and proportional hazard models. NFT BART has three key features: (1) a BART prior for the mean function of the event time logarithm; (2) a heteroskedastic BART prior to deduce a covariate-dependent variance function; and (3) a flexible nonparametric error distribution using Dirichlet process mixtures (DPM). Our proposed approach widens the scope of hazard shapes including nonproportional hazards, can be scaled up to large sample sizes, naturally provides estimates of uncertainty via the posterior and can be seamlessly employed for variable selection. We provide convenient, user-friendly, computer software that is freely available as a reference implementation. Simulations demonstrate that NFT BART maintains excellent performance for survival prediction especially when AFT assumptions are violated by heteroskedasticity. We illustrate the proposed approach on a study examining predictors for mortality risk in patients undergoing hematopoietic stem cell transplant (HSCT) for blood-borne cancer, where heteroskedasticity and nonproportional hazards are likely present.

AbstractMethods based on binary trees play a fundamental role in modern data science. In this article, we give a very focused review of basic Bayesian approaches to tree modeling. Bayesian approaches have some fundamental advantages. Complex models are enhanced with meaningful prior specifications, and Markov chain Monte Carlo provides a framework for useful stochastic search of the model space along with some sense of uncertainty. Bayesian approaches require a specification of a prior in tree space and computation of a high and variable dimension posterior. We provide some computational details that are not readily available in the literature. We also review a few more recent extensions of the basic approaches to illustrate the power and potential of the overall approach.

State–space models are commonly used in the engineering, economic, and statistical literature. They are flexible and encompass many well-known statistical models, including random coefficient autoregressive models and dynamic factor models. Bayesian analysis of state–space models has attracted much interest in recent years. However, for large scale models, prior specification becomes a challenging issue in Bayesian inference. In this paper, we propose a flexible prior for state–space models. The proposed prior is a mixture of four commonly entertained models, yet achieving parsimony in high-dimensional systems. Here “parsimony” is represented by the idea that, in a large system, some states may not be time-varying. Our prior for the state–space component’s standard deviation is capable to accommodate different scenarios. Simulation and simple examples are used throughout this paper to demonstrate the performance of the proposed prior. As an application, we consider the time-varying conditional covariance matrices of daily log returns of the components of the S&P 100 index, leading to a state–space model with roughly five thousand time-varying states. Our model for this large system enables us to use parallel computing.

For the discovery of regression relationships between Y and a large set of p potential predictors x(1), ... , x(p), the flexible nonparametric nature of BART (Bayesian Additive Regression Trees) allows for a much richer set of possibilities than restrictive parametric approaches. However, subject matter considerations sometimes warrant a minimal assumption of monotonicity in at least some of the predictors. For such contexts, we introduce mBART, a constrained version of BART that can flexibly incorporate monotonicity in any predesignated subset of predictors using a multivariate basis of monotone trees, while avoiding the further confines of a full parametric form. For such monotone relationships, mBART provides (i) function estimates that are smoother and more interpretable, (ii) better out-of-sample predictive performance, and (iii) less post-data uncertainty. While many key aspects of the unconstrained BART model carry over directly to mBART, the introduction of monotonicity constraints necessitates a fundamental rethinking of how the model is implemented. In particular, the original BART Markov Chain Monte Carlo algorithm relied on a conditional conjugacy that is no longer available in a monotonically constrained space. Various simulated and real examples demonstrate the wide ranging potential of mBART.

In this article, we introduce the BART R package which is an acronym for Bayesian additive regression trees. BART is a Bayesian nonparametric, machine learning, ensemble predictive modeling method for continuous, binary, categorical and time-to-event outcomes. Furthermore, BART is a tree-based, black-box method which fits the outcome to an arbitrary random function, f , of the covariates. The BART technique is relatively computationally efficient as compared to its competitors, but large sample sizes can be demanding. Therefore, the BART package includes efficient state-of-the-art implementations for continuous, binary, categorical and time-to-event outcomes that can take advantage of modern off-the-shelf hardware and software multi-threading technology. The BART package is written in C++ for both programmer and execution efficiency. The BART package takes advantage of multi-threading via forking as provided by the parallel package and OpenMP when available and supported by the platform. The ensemble of binary trees produced by a BART fit can be stored and re-used later via the R predict function. In addition to being an R package, the installed BART routines can be called directly from C++. The BART package provides the tools for your BART toolbox.

We provide a new flexible framework for inference with the instrumental variable model. Rather than using linear specifications, functions characterizing the effects of instruments and other explanatory variables are estimated using machine learning via Bayesian Additive Regression Trees (BART). Error terms and their distribution are inferred using Dirichlet Process mixtures. Simulated and real examples show that when the true functions are linear, little is lost. But when nonlinearities are present, dramatic improvements are obtained with virtually no manual tuning.

PURPOSE Donor selection practices for matched unrelated donor (MUD) hematopoietic cell transplantation (HCT) vary, and the impact of optimizing donor selection in a patient-specific way using modern machine learning (ML) models has not been studied. METHODS We trained a Bayesian ML model in 10,318 patients who underwent MUD HCT from 1999 to 2014 to provide patient- and donor-specific predictions of clinically severe (grade 3 or 4) acute graft-versus-host disease or death by day 180. The model was validated in 3,501 patients from 2015 to 2016 with archived records of potential donors at search. Donor selection optimizing predicted outcomes was implemented over either an unlimited donor pool or the donors in the search archives. Posterior mean differences in outcomes from optimal donor selection versus actual practice were summarized per patient and across the population with 95% intervals. RESULTS Event rates were 33% (training) and 37% (validation). Among donor features, only age affected outcomes, with the effect consistent regardless of patient features. The median (interquartile range) difference in age between the youngest donor at search and the selected donor was 6 (1-10) years, whereas the number of donors per patient younger than the selected donor was 6 (1-36). Fourteen percent of the validation data set had an approximate 5% absolute reduction in event rates from selecting the youngest donor at search versus the actual donor used, leading to an absolute population reduction of 1% (95% interval, 0 to 3). CONCLUSION We confirmed the singular importance of selecting the youngest available MUD, irrespective of patient features, identified potential for improved HCT outcomes by selecting a younger MUD, and demonstrated use of novel ML models transferable to optimize other complex treatment decisions in a patient-specific way.

Bayesian Additive Regression Trees (BART) has emerged as a highly effective Bayesian approach to ensemble modeling with many binary trees. The BART Markov Chain Monte Carlo (MCMC) algorithm provides effective stochastic search in a complex model space and Bayesian uncertainty. As is the case with many modern approaches, the overall complexity of the model makes interpretation difficult. In practice, investigators often wish to know what predictor variables are important or, more generally, which roles variables play in the model. In this chapter we review some approaches for understanding how variables enter the BART model. We present simple ways to find out which variables are most important, which pairs of variables interact in the model, and which subsets of variables allow us to approximate the full information inference according to a user defined metric. In all cases, our approaches are based on post processing the output from basic BART modeling, naturally capturing the uncertainty by the usual MCMC variation in a straightforward way.

BART (Bayesian Additive Regression Trees) has become increasingly popular as a flexible and scalable nonparametric regression approach for modern applied statistics problems. For the practitioner dealing with large and complex nonlinear response surfaces, its advantages include a matrix-free formulation and the lack of a requirement to prespecify a confining regression basis. Although flexible in fitting the mean, BART has been limited by its reliance on a constant variance error model. This homoscedastic assumption is unrealistic in many applications. Alleviating this limitation, we propose HBART, a nonparametric heteroscedastic elaboration of BART. In BART, the mean function is modeled with a sum of trees, each of which determines an additive contribution to the mean. In HBART, the variance function is further modeled with a product of trees, each of which determines a multiplicative contribution to the variance. Like the mean model, this flexible, multidimensional variance model is entirely nonparametric with no need for the prespecification of a confining basis. Moreover, with this enhancement, HBART can provide insights into the potential relationships of the predictors with both the mean and the variance. Practical implementations of HBART with revealing new diagnostic plots are demonstrated with simulated and real data on used car prices, fishing catch production and alcohol consumption.

Many time-to-event studies are complicated by the presence of competing risks. Such data are often analyzed using Cox models for the cause-specific hazard function or Fine and Gray models for the subdistribution hazard. In practice, regression relationships in competing risks data are often complex and may include nonlinear functions of covariates, interactions, high-dimensional parameter spaces and nonproportional cause-specific, or subdistribution, hazards. Model misspecification can lead to poor predictive performance. To address these issues, we propose a novel approach: flexible prediction modeling of competing risks data using Bayesian Additive Regression Trees (BART). We study the simulation performance in two-sample scenarios as well as a complex regression setting, and benchmark its performance against standard regression techniques as well as random survival forests. We illustrate the use of the proposed method on a recently published study of patients undergoing hematopoietic stem cell transplantation.