The presence of covariate shift between training and test datasets, coupled with model misspecification, can lead to instability in regression predictions across diverse datasets. Meanwhile, training complex models with massive data imposes significant computational burden. In this article, we present a novel model-free subsampling algorithm for stable prediction, which employs uniform design and confounder balancing methods. Our subsampling algorithm aims to find the nearest neighbor subsampling points of uniform design with the goal of minimizing global stability loss, thereby reducing the data volume while achieving stable predictions. Theoretic analyses show that the uniform measure minimizes the maximum integrated mean square error (MIMSE) and the global stability loss evaluates the independence among variables in each candidate MIMSE-optimal subsampled sets. Simulation studies conducted on synthetic datasets, as well as applications on real datasets, demonstrate the superiority of our proposed method under model misspecification and covariate shift.

Tree ensemble models like random forests and gradient boosting machines are widely used in machine learning due to their excellent predictive performance. However, a high-performance ensemble consisting of a large number of decision trees lacks sufficient transparency and explainability. In this paper, we demonstrate that when shallow decision trees are used as base learners, the ensemble learning algorithms can not only become inherently interpretable subject to an equivalent representation as the generalized additive models but also sometimes lead to better generalization performance. First, an interpretation algorithm is developed that converts the tree ensemble into the functional ANOVA representation with inherent interpretability. Second, two strategies are proposed to further enhance the model interpretability, i.e., by adding constraints in the model training stage and post-hoc effect pruning. Experiments on simulations and real-world datasets show that our proposed methods offer a better trade-off between model interpretation and predictive performance, compared with its counterpart benchmarks.

The additive index models (AIMs) can be viewed as a kind of artificial neural networks based on nonparametric activation or so-called ridge functions. Recently, they are shown to achieve enhanced explainability after incorporating various interpretability constraints. However, the training of AIMs by either the backfitting algorithm or the joint stochastic optimization is known to be very slow for especially high dimensional inputs. In this article, we propose a novel sequential approach based on the celebrated Stein's lemma. The proposed SeqStein method can successfully decouple the training of AIMs into two separable steps, namely, the following: 1) Stein's estimation of the projection indices and 2) nonparametric estimation of ridge functions using the smoothing splines. We show through numerical experiments that the SeqStein algorithm is not only more efficient for training AIMs, but also inclined to produce more interpretable models that have smooth ridge functions with sparse and nearly orthogonal projection indices.

Subsampling plays a crucial role in tackling problems associated with the storage and statistical learning of massive datasets. However, most existing subsampling methods are model-based, which means their performances can drop significantly when the underlying model is misspecified. Such an issue calls for model-free subsampling methods that are robust under diverse model specifications. Recently, several model-free subsampling methods have been developed. However, the computing time of these methods grows explosively with the sample size, making them impractical for handling massive data. In this article, an efficient model-free subsampling method is proposed, which segments the original data into some regular data blocks and obtains subsamples from each data block by the data-driven subsampling method. Compared with existing model-free subsampling methods, the proposed method has a significant speed advantage and performs more robustly for datasets with complex underlying distributions. As demonstrated in simulation experiments, the proposed method is an order of magnitude faster than other commonly used model-free subsampling methods when the sample size of the original dataset reaches the order of 107. Moreover, simulation experiments and case studies show that the proposed method is more robust than other model-free subsampling methods under diverse model specifications and subsample sizes.

This study leverages novel machine learning techniques to quantify the complex empirical relationship between mortgage default and its drivers. The primary model employed is the authors’ newly developed shape-constrained GAMI-Net, which introduces lattice-function-based main effects and pairwise interactions that take user-defined shape constraints. Their approach of adding shape constraints to a lattice module enhances the interpretability and applicability of the model in real-world scenarios. The authors compare the performance of shape-constrained GAMI-Net with alternative machine learning and traditional statistical methods using Freddie Mac’s publicly available mortgage dataset. The results demonstrate competitive predictive performance and high interpretability for the shape-constrained GAMI-Net model.

In this paper, a novel single-index model tree (SIMTree) is proposed. It adopts the recursive partitioning strategy and each data segment is modeled by a single-index model (SIM), which is a flexible extension of linear regression with non-parametric link functions. The proposed SIMTree has two major advantages: a) with only a few leaf nodes, it can achieve competitive predictive performance compared to complicated black-box models; b) SIMs fitted on each local data segment are intrinsically interpretable. However, using conventional techniques to build such a SIMTree can be extremely time-consuming. SIM estimation typically involves iterative optimization via Newton-type algorithms; such a resource-intensive estimation procedure is repeatedly used for fitting leaf node SIMs and the search of optimal splits. To make the computation burden affordable, an effective training algorithm is proposed as enabled by the efficient utilization of Stein's lemma and several accelerating strategies in the tree construction algorithm. Moreover, a new Python package simtree is developed with elegant visualization modules that can further facilitate the model interpretation. Numerical results on extensive regression datasets show that SIMTree is an accurate and interpretable machine learning model.

PiML (read π-ML, /`pai`em`el/) is an integrated and open-access Python toolbox for interpretable machine learning model development and model diagnostics. It is designed with machine learning workflows in both low-code and high-code modes, including data pipeline, model training and tuning, model interpretation and explanation, and model diagnostics and comparison. The toolbox supports a growing list of interpretable models (e.g. GAM, GAMI-Net, XGB1/XGB2) with inherent local and/or global interpretability. It also supports model-agnostic explainability tools (e.g. PFI, PDP, LIME, SHAP) and a powerful suite of model-agnostic diagnostics (e.g. weakness, reliability, robustness, resilience, fairness). Integration of PiML models and tests to existing MLOps platforms for quality assurance are enabled by flexible high-code APIs. Furthermore, PiML toolbox comes with a comprehensive user guide and hands-on examples, including the applications for model development and validation in banking. The project is available at https://github.com/SelfExplainML/PiML-Toolbox.

•A novel explainable neural network is proposed for modeling main effects and structured interactions.•The GAMI-Net is a disentangled feedforward network with multiple additive subnetworks.•GAMI-Net takes into account three interpretability constraints: sparsity, heredity, marginal clarity.•An adaptive training algorithm is developed for training GAMI-Net efficiently.•GAMI-Net enjoys superior interpretability and outperforms benchmark methods.

Prediction accuracy and model explainability are the two most important objectives when developing machine learning algorithms to solve real-world problems. Neural networks are known to possess good prediction performance but suffer from a lack of model interpretability. In this article, we propose to enhance the explainability of neural networks through the following architecture constraints: 1) sparse additive subnetworks; 2) projection pursuit with orthogonality constraint; and 3) smooth function approximation. It leads to an enhanced explainable neural network (ExNN) with a superior balance between prediction performance and model interpretability. We derive sufficient identifiability conditions for the proposed ExNN model. The multiple parameters are simultaneously estimated by a modified minibatch gradient descent method based on the backpropagation algorithm for calculating the derivatives and the Cayley transform for preserving the projection orthogonality. Through simulation study under six different scenarios, we compare the proposed method to several benchmarks, including least absolute shrinkage and selection operator, support vector machine, random forest, extreme learning machine, and multilayer perceptron. It is shown that the proposed ExNN model keeps the flexibility of pursuing high prediction accuracy while attaining improved interpretability. Finally, a real data example is employed as a showcase application.

Hyperparameter optimization (HPO) plays a central role in the automated machine learning (AutoML). It is a challenging task as the response surfaces of hyperparameters are generally unknown, hence essentially a global optimization problem. This paper reformulates HPO as a computer experiment and proposes a novel sequential uniform design (SeqUD) strategy with three-fold advantages: a) the hyperparameter space is adaptively explored with evenly spread design points, without the need of expensive meta-modeling and acquisition optimization; b) the batch-by-batch design points are sequentially generated with parallel processing support; c) a new augmented uniform design algorithm is developed for the efficient real-time generation of follow-up design points. Extensive experiments are conducted on both global optimization tasks and HPO applications. The numerical results show that the proposed SeqUD strategy outperforms benchmark HPO methods, and it can be therefore a promising and competitive alternative to existing AutoML tools.

Network initialization is the first and critical step for training neural networks. In this paper, we propose a novel network initialization scheme based on the celebrated Stein's identity. By viewing multi-layer feedforward neural networks as cascades of multi-index models, the projection weights to the first hidden layer are initialized using eigenvectors of the cross-moment matrix between the input's second-order score function and the response. The input data is then forward propagated to the next layer and such a procedure can be repeated until all the hidden layers are initialized. Finally, the weights for the output layer are initialized by generalized linear modeling. Such a proposed SteinGLM method is shown through extensive numerical results to be much faster and more accurate than other popular methods commonly used for training neural networks.

In this paper, we make a comprehensive review of the challenging task of hyperparameter optimization in automated machine learning.The commonly used hyperparameter optimization methods include single-shot sampling strategies, e.g., grid search, random search and sequential strategies where new trials are gradually augmented based on existing information, including Bayesian optimization, evolutionary algorithms, reinforcement learning-based methods.We find the sequential number-theoretic optimization (SNTO) algorithm proposed by Kai-Tai Fang and Yuan Wang in 1990 can also be applied in hyperparameter optimization, where sequential uniform designs are utilized to search the global optima in complex response surfaces.For illustration, various hyperparameter optimization methods are tested with two widely-used machine learning models including the support vector machine (SVM) and extreme gradient boosting (XGBoost), on two classical binary classification datasets.By analyzing the experimental results, we find a modified SNTO algorithm is quite promising in the hyperparameter optimization task.

In recent years, the field of recommendation systems has attracted increasing attention to developing predictive models that provide explanations of why an item is recommended to a user. The explanations can be either obtained by post-hoc diagnostics after fitting a relatively complex model or embedded into an intrinsically interpretable model. In this paper, we propose the explainable recommendation systems based on a generalized additive model with manifest and latent interactions (GAMMLI). This model architecture is intrinsically interpretable, as it additively consists of the user and item main effects, the manifest user-item interactions based on observed features, and the latent interaction effects from residuals. Unlike conventional collaborative filtering methods, the group effect of users and items are considered in GAMMLI. It is beneficial for enhancing the model interpretability, and can also facilitate the cold-start recommendation problem. A new Python package GAMMLI is developed for efficient model training and visualized interpretation of the results. By numerical experiments based on simulation data and real-world cases, the proposed method is shown to have advantages in both predictive performance and explainable recommendation.

The deep neural networks (DNNs) have achieved great success in learning complex patterns with strong predictive power, but they are often thought of as "black box" models without a sufficient level of transparency and interpretability. It is important to demystify the DNNs with rigorous mathematics and practical tools, especially when they are used for mission-critical applications. This paper aims to unwrap the black box of deep ReLU networks through local linear representation, which utilizes the activation pattern and disentangles the complex network into an equivalent set of local linear models (LLMs). We develop a convenient LLM-based toolkit for interpretability, diagnostics, and simplification of a pre-trained deep ReLU network. We propose the local linear profile plot and other visualization methods for interpretation and diagnostics, and an effective merging strategy for network simplification. The proposed methods are demonstrated by simulation examples, benchmark datasets, and a real case study in home lending credit risk assessment.

The prediction of interval-valued data is a challenging task as the predicted lower bounds of intervals should not cross over the corresponding upper bounds. In this paper, a regularized artificial neural network (RANN) is proposed to address this difficult problem. It provides a flexible trade-off between prediction accuracy and interval crossing. Compared to existing hard-constrained methods, the RANN has the advantage that it does not necessarily reduce the prediction accuracy while preventing interval crossing. Extensive experiments are conducted based on both simulation and real-life datasets, with comparison to multiple traditional models, including the linear constrained center and range method, the least absolute shrinkage and selection operator-based interval-valued regression, the nonlinear interval kernel regression, the interval multi-layer perceptron and the multi-output support vector regression. Experimental results show that the proposed RANN model is an effective tool for interval-valued data prediction tasks with high prediction accuracy.

The rapid development of big data technologies and the Internet provides a rich mine of online big data (e.g., trend spotting) that can be helpful in predicting oil consumption — an essential but uncertain factor in the oil supply chain. An online big data-driven oil consumption forecasting model is proposed that uses Google trends, which finely reflect various related factors based on a myriad of search results. This model involves two main steps, relationship investigation and prediction improvement. First, cointegration tests and a Granger causality analysis are conducted in order to statistically test the predictive power of Google trends, in terms of having a significant relationship with oil consumption. Second, the effective Google trends are introduced into popular forecasting methods for predicting both oil consumption trends and values. The experimental study of global oil consumption prediction confirms that the proposed online big-data-driven forecasting work with Google trends improves on the traditional techniques without Google trends significantly, for both directional and level predictions.

To guarantee stable quantile estimations even for noisy data, a novel loss function and novel quantile estimators are developed, by introducing the effective concept of orthogonal loss considering the noise in both response and explanatory variables. In particular, the pinball loss used in classical quantile estimators is improved into novel orthogonal pinball loss (OPL) by replacing vertical loss by orthogonal loss. Accordingly, linear quantile regression (QR) and support vector machine quantile regression (SVMQR) can be respectively extended into novel OPL-based QR and OPL-based SVMQR models. The empirical study on 10 publicly available datasets statistically verifies the superiority of the two OPL-based models over their respective original forms in terms of prediction accuracy and quantile property, especially for extreme quantiles. Furthermore, the novel OPL-based SVMQR model with both OPL and artificial intelligence (AI) outperforms all benchmark models, which can be used as a promising quantile estimator, especially for noisy data.

Due to the uncertainty in oil markets, this paper proposes a novel approach for oil purchasing and distribution optimization by incorporating price and demand prediction, i.e., the prediction-based oil purchasing-and-distribution optimization model. In particular, the proposed method bridges the latest information technology and decision-making technique by introducing the recently proposed information technology (i.e., extreme learning machine (ELM)) into the oil purchasing-and-distribution optimization model. Two main steps are involved: market prediction and planning optimization in the proposed model. In market prediction, the ELM technique is employed to provide fast training time and accurate forecasting results for oil prices and demands. In planning optimization, two objectives of general profit maximization and inventory risk minimization are considered; and the most popular multi-objective evolutionary algorithm (MOEA), nondominated sorting genetic algorithm II (NSGA-II), is implemented to search approximate Pareto optimal solutions. For illustration and verification, the motor gasoline market in the US is focused on as the study sample, and the experimental results demonstrate the superiority of the proposed prediction-based optimization approach over its benchmark models (without market prediction and/or planning optimization), in terms of the highest profit and the lowest risk.

To achieve high assessment accuracy for credit risk, a novel multistage deep belief network (DBN) based extreme learning machine (ELM) ensemble learning methodology is proposed. In the proposed methodology, three main stages, i.e., training subsets generation, individual classifiers training and final ensemble output, are involved. In the first stage, bagging sampling algorithm is applied to generate different training subsets for guaranteeing enough training data. Second, the ELM, an effective AI forecasting tool with the unique merits of time-saving and high accuracy, is utilized as the individual classifier, and diverse ensemble members can be accordingly formulated with different subsets and different initial conditions. In the final stage, the individual results are fused into final classification output via the DBN model with sufficient hidden layers, which can effectively capture the valuable information hidden in ensemble members. For illustration and verification, the experimental study on one publicly available credit risk dataset is conducted, and the results show the superiority of the proposed multistage DBN-based ELM ensemble learning paradigm in terms of high classification accuracy.

一套完善的风险评估体系对于中国企业的国际化过程有着重要的意义.对于业务投资、运营等各个环节的重要风险点进行评估与预警,可以提前防范可能的风险,减少不必要的损失.本文对适合中国企业国际化风险评估的各种方法进行概括梳理,按定性评估方法、定量评估方法和定性定量相结合的评估方法分别进行阐述,并总结与归纳各种方法的优缺点以及适用情景.