In recent years, the problem of classification with imbalanced data has been growing in popularity in the data-mining and machine-learning communities due to the emergence of an abundance of imbalanced data in many fields. In this chapter, we compare the performance of six classification methods on an imbalanced dataset under the influence of four resampling techniques. These classification methods are the random forest, the support vector machine, logistic regression, k-nearest neighbor (KNN), the decision tree, and AdaBoost. Our study has shown that all of the classification methods have difficulty when working with the imbalanced data, with the KNN performing the worst, detecting only 27.4% of the minority class. However, with the help of resampling techniques, all of the classification methods experience improvement on overall performances. In particular, the Random Forest, in combination with the random over-sampling technique, performs the best, achieving 82.8% balanced accuracy (the average of the true-positive rate and true-negative rate). Keywords Imbalanced data Resampling Under-sampling Classification Random Forest Rare event detection Imbalanced learning Citation Nguyen, S., Niu, G., Quinn, J., Olinsky, A., Ormsbee, J., Smith, R.M. and Bishop, J. (2019), "Detecting Non-injured Passengers and Drivers in Car Accidents: A New Under-resampling Method for Imbalanced Classification", Advances in Business and Management Forecasting (Advances in Business and Management Forecasting, Vol. 13), Emerald Publishing Limited, Bingley, pp. 93-105. https://doi.org/10.1108/S1477-407020190000013011 Publisher: Emerald Publishing Limited Copyright © 2019 Emerald Publishing Limited Introduction The emergence of Big Data, where data grow at an increasing rate in volume, velocity, and variety (Zikopoulos, Eaton, deRoos, Deutsch, & Lapis, 2011), in many disciplines in the past decades, poses many challenges for the fields of statistics, machine learning, and data mining. One of these significant issues is the problem of classification with imbalanced data, where the values for the dependent variable (target variable, class variable) are distributed highly unevenly. For example, these values (observations) might occur less than 1% of the time. In the past years, this topic has drawn a large amount of attention in machine learning, statistics, and other data-related communities (Krawczyk, 2016). In imbalanced data classification, the minority class (positive class) is the class preferred to be correctly classified and usually costs more when misclassified, that is, a false negative is more severe than a false positive. For instance, in cancer detection, wrongly diagnosing a cancer patient may result in the loss of life. The main trouble with classifying imbalanced data using traditional classification algorithms is that the true-positive rate, or sensitivity, is often very low. This is due to the fact that most of these traditional algorithms assume the values for the target variable is distributed equally. Thus, when applied to imbalanced data, the algorithms tend to favor the majority class, resulting in a high true-negative rate and low true-positive rate. To improve the true-positive rate while maintaining a good overall performance in imbalanced classification, a number of techniques have been developed. These techniques mainly fall into three categories: resampling method, modifying the traditional algorithm, and a combination of these two. While the goal of resampling is to balance the data, algorithmic methods reconstruct the objective functions in classification methods in order to give higher weights for the minority class. Some of the most popular resampling techniques include the following: synthetic minority over-sampling technique (SMOTE) (Chawla, Bowyer, Hall, & Kegelmeyer, 2002), random over-sampling examples (ROSE) (Menardi & Torelli, 2014), and borderline-SMOTE (Han, Wang, & Mao, 2005). Examples of classification methods that have been modified to work with imbalanced datasets include the following: Random Forest (Chen, Liaw, & Breiman, 2004) and the decision tree (Cieslak & Chawla, 2008) In this chapter, we compare the performances of six popular classification methods (the decision tree, the Random Forest, logistic regression, SVM, AdaBoost, and k-nearest neighbor (KNN)) coupled with four resampling techniques (random under-resampling, random over-resampling, SMOTE, and ROSE) on a car accident dataset. We then propose an under-resampling technique that can further improve the performance of the best model out of the six – the Random Forest. Data Our data are obtained from the accident dataset from the fatality analysis reporting system (FARS) and pertains to 2015. The dataset has 80,565 observations, where each observation represents a person involved in the car accident, and 25 variables, containing information such as the person’s sex, age, the location of the accident, and the time and date it took place. Our target variable contains the aftermath injury information. All observations are grouped into two classes: “Injured” class, including people who were injured in the accident, and “Not Injured” class. Thus, the target variable either takes the value of “Injured” or “Not Injured.” The “Not Injured” class accounts for 24.4%, which can be considered as the minority class as well as the positive class. We provide several graphs below representing the relation between the target variable and a few other input variables. From the graph shown in Fig. 1, we can see that the proportion of injured people in the group of people who drank alcohol before the accidents is greater than that of in the group of people who did not drink. Also, Fig. 2 suggests that the variable sex and the target variable seem to be independent of each other as given the information about sex, the proportion of injured people does not change much.Opens in a new window.Fig. 1. Distribution of the Target Variable in Different Categories of Drinking. Opens in a new window.Fig. 2. Distribution of the Target Variable with Respect to Gender. We also observe from Fig. 3 that the distribution of age in both classes is quite similar, with the majority of people being young (less than 30). Fig. 4 indicates that there seems to be more injuries in accidents occurring from midnight to noon than at other times.Opens in a new window.Fig. 3. Approximate Density Distribution of People’s Age Who Were in Fatal Car Accidents. Opens in a new window.Fig. 4. Approximated Density Distribution of the Hour When Accidents Occurred. Method In this section, we describe the classification methods and resampling methods used in our computation. We also introduce our proposed method to further improve the Random Forest, which is the “best” model (the model that gives the greatest averaged balanced accuracy) out of the six classification models. To test the performance of a classification model, we partition the original data into two datasets: training dataset (70%) and testing dataset (30%). The model is then built using the training dataset and once it is built, it is run and its predictions are checked on the testing dataset. There are missing values in the data, which we impute by the predictive mean-matching method (Rubin, 1986). All the calculations are done in R. The six classification methods we use are the decision tree, the Random Forest, logistic regression, KNN, the SVM, and AdaBoost. Decision Tree The decision tree or classification and regression tree (CART) (Breiman, Friedman, Olshen, & Stone, 1983) is a machine-learning algorithm that can be used with both a categorical target (classification tree) and a continuous target (regression tree). The decision tree structure is a flow chart where at each of its nodes, the incoming data are partitioned into multiple subsets based on a splitting rule. The final nodes, where the splitting stops, are called the leaves and are used for prediction. The tree is usually fully grown to obtain the most complicated tree and then pruned down to a simpler tree to avoid the over-fitting issue. In this calculation, we use a binary tree (in which a node can only have two branches) with the splitting rule using the Gini Index. We use package “rpart” (Recursive PARTition) for the computation. Random Forest The Random Forest is an ensemble method, which is the method that combines the results of multiple other methods. In the Random Forest, a set of decision trees is built on a subset of the original data. Each tree does not use all the variables but only a randomly selected subset of them to decide the split points. The Random Forest, then, makes predictions by majority voting between the trees. For instance, if three out of five trees of a Random Forest predict “Not Injured,” then the Random Forest will also predict “Not Injured.” Two main hyperparameters (unknown quantities that are determined by users) of the Random Forest are the number of trees and the number of variables selected at each node of a tree. In our calculations, we set the number of trees to 30 and the number of variables at each node to 2. Our computations of the Random Forest are carried out using the library “ranger” in R. Logistic Regression Logistic regression is one of the older and well-understood classification methods developed by David Cox in 1958 (Cox, 1958). Logistic regression is a linear classification method, that is, it forms a hyperplane that separates the negative and positive observations. The equation connects p , the probability of the target variable Y being positive, and the input variable vector, X , as follows. logit ( p ) = log p 1 − p = a + b X This equation can be written in the form of a logistic function, the inverse of the logit function. To find the parameters (the slope and the intercept), maximum-likelihood methods are often used. In our study, we use the package “glm” (generalized linear models) to compute logistic regression. K-Nearest Neighbor (KNN) KNN is one of the simplest supervised machine-learning algorithms. For classification, the prediction of a new observation is the majority of its KNN s. For example, if three out of five of the closest points of a new observation are classified as positive, then the 5-nearest neighbor will predict the new observation also as positive. To measure the “nearest,” a variety of distance measures can be applied, with the most popular one being the Euclidean distance. The hyperparameter for KNN is k , the number of nearest neighbors. We set k = 5 and use function KNN in the library class in R for our computation. Support Vector Machine (SVM) SVM was proposed by Vapnik in 1963 for the linear case (Vapnik & Lerner, 1963). The non-linear kernel version of SVM and the soft-margin SVM were introduced in 1992 (Boser, Guyon, & Vapnik, 1992) and in 1995 (Cortes & Vapnik, 1995). If the data are linearly separable, that is, there exists a hyperplane that separates the positive and negative classes, SVM looks for a hyperplane that separates the two classes with a maximum margin, where the margin is the distance of the closest point to the hyperplane. If the data are non-linearly separable, SVM can “soften” the margin to allow data points to cross the margin. The number of errors caused by data points positioned in a wrong region is controlled by the cost parameter. One can also use the “kernel trick” to transform the data to be linearly separable. We implement SVM using the “SVM” function in the “e1071” library with the radial basic function kernel and the cost parameter set to 1. AdaBoost AdaBoost, or Adaptive Boosting, is designed to improve the performance of a “weak” classification algorithm. The key idea of AdaBoost is to pay more attention to misclassified observations by giving them higher weights than correctly classified observations. Based on this idea, AdaBoost generates a sequence of classification methods where each method in the sequence is focused more on learning the error of its previous classification. All of the classification methods in the sequence, then, contribute to the final prediction of AdaBoost. In our computation, we use AdaBoost to boost a decision tree. We set the number of iterations to 100, that is, the classification sequence contains 100 decision trees. Resampling Method One of the main reasons for resampling the imbalanced data is to balance it so that regular classification methods can work with these data. Here, we consider four of the most frequently used resampling methods: randomly under-resampling (RUS), randomly over-resampling (ROS), SMOTE, and ROSE. Resampling methods fall into two categories: under-sampling and over-sampling. While under-sampling aims to reduce the size of the majority/negative class down to that of the minority class, over-sampling methods add observations to the minority/positive class. In the four methods, only RUS is an under-resampling method, and the other three belong to the class of over-resampling methods. Due to their simplicity in concept and computation, RUS and ROS are very frequently used resampling methods. In RUS, the number of negative examples is removed randomly to balance the negative class with the positive class. In ROS, positive examples are added by bootstrapping with replacement on the original positive class. While the downside of RUS is removing the original data, which may contain important information to build a prediction at the later stage, the downside of ROS is increasing the computation intensity, which could be severe if the sample size is extremely large. SMOTE (Chawla et al., 2002) and ROSE (Menardi & Torelli, 2014) artificially generated new positive examples. In SMOTE, new examples are created from a point, for example, point A and its neighbors. New points are random points lying on the lines connecting A with its neighbors. In ROSE, the generation of new examples from the minority class corresponds to the generation of data from the kernel conditional density estimate of the minority class. In all four resampling techniques, we balance the data so that each of the classes accounts for approximately 50%. In SMOTE, we choose the number of KNNs to be 5. Selective Under-resampling (SUR) In AdaBoost, the correctly classified observations are considered “easier” observations to classify and are given smaller weights in the next iteration. Inspired by this idea, we propose an under-sampling procedure that eliminates “easier” negative observations before applying the traditional under-sampling method. The “easiness” of a negative example is measured by how close its predicted probability is to 0. By this measurement, a negative observation with predicted probability 0 would be the “easiest” negative example, whereas a negative example with predicted probability 1 would be considered the “hardest” negative example. Note also that all the negative examples with predicted probability greater than 0.5 are misclassified observations. To compute the predicted probability for the negative observations, first we run the Random Forest on the data. Here, we choose the Random Forest to implement this approach because it gives the best performance out of the six tested methods. Our under-resampling and modeling works as follows: Step 1: run the Random Forest on the training dataset to obtain the predicted probabilities for all of the observations; Step 2: eliminate all negative examples that have a predicted probability smaller than p ; Alternatively, eliminate the top p % of observations with the lowest predicted probabilities. Obtain a subset S of the negative class; Step 3: randomly under-resample k times from S , to obtain balanced dataset B 1 , B 2 … B k ; Step 4: run the Random Forest on B 1 , B 2 … B k to build k Random Forests; and Step 5: combine the k Random Forests in Step 4 by voting to obtain the final model. In our implementation, we eliminate 15% of majority observations with the lowest predicted probabilities and we choose k = 7 . Results We report four different metrics of the classification models: (1) the sensitivity, or the true-positive rate; (2) the specificity, or the true-negative rate; (3) the accuracy, which is the ratio between the number of corrected predictions over the total predictions; and (4) the balanced accuracy, which is the average of the sensitivity and specificity. Since the goal of imbalanced data classification is to increase the sensitivity without reducing significantly the specificity, we use the balanced accuracy as the main metric to measure the overall performance of the classification models and resampling techniques. Our detailed results are reported in Tables 1 to 6 and illustrated in Figs 5, 6, and 7. Table 1. Random Forest Performance Metrics on Original Dataset and Five Resampling Methods. Original Under Over SMOTE ROSE SUR Accuracy 0.877 0.842 0.88 0.873 0.842 0.87 Sensitivity 0.605 0.791 0.726 0.723 0.774 0.774 Specificity 0.965 0.858 0.93 0.922 0.864 0.901 Balanced accuracy 0.785 0.825 0.828 0.822 0.819 0.837 Table 2. Performance Metrics of Six Classification Methods Running on the Original Dataset. Decision Tree Random Forest Logistic Regression KNN SVM AdaBoost Accuracy 0.877 0.877 0.879 0.739 0.877 0.88 Sensitivity 0.62 0.605 0.63 0.274 0.612 0.633 Specificity 0.961 0.965 0.959 0.889 0.963 0.96 Balanced accuracy 0.79 0.785 0.795 0.581 0.787 0.796 Table 3. Performance Metrics of Six Classification Methods Running on the Under-resampling Dataset. Decision Tree Random Forest Logistic Regression KNN SVM AdaBoost Accuracy 0.835 0.842 0.843 0.606 0.857 0.833 Sensitivity 0.753 0.791 0.774 0.637 0.751 0.801 Specificity 0.862 0.858 0.866 0.595 0.891 0.844 Balanced accuracy 0.807 0.825 0.82 0.616 0.821 0.822 Table 4. Performance Metrics of Six Classification Methods Running on the Over-resampling Dataset. Decision Tree Random Forest Logistic Regression KNN SVM AdaBoost Accuracy 0.836 0.88 0.847 0.588 0.864 0.84 Sensitivity 0.75 0.726 0.769 0.649 0.753 0.785 Specificity 0.864 0.93 0.872 0.568 0.9 0.857 Balanced accuracy 0.807 0.828 0.821 0.608 0.826 0.821 Table 5. Performance Metrics of Six Classification Methods Running on the SMOTE Dataset. Decision Tree Random Forest Logistic Regression KNN SVM AdaBoost Accuracy 0.829 0.873 0.833 0.597 0.866 0.86 Sensitivity 0.737 0.723 0.773 0.668 0.735 0.734 Specificity 0.859 0.922 0.852 0.574 0.909 0.901 Balanced accuracy 0.798 0.822 0.813 0.621 0.822 0.817 Table 6. Performance Metrics of Six Classification Methods Running on the ROSE Dataset. Decision Tree Random Forest Logistic Regression KNN SVM AdaBoost Accuracy 0.829 0.842 0.846 0.596 0.86 0.838 Sensitivity 0.729 0.774 0.763 0.572 0.746 0.788 Specificity 0.862 0.864 0.873 0.604 0.897 0.854 Balanced accuracy 0.795 0.819 0.818 0.588 0.821 0.821 Opens in a new window.Fig. 5. Balanced Accuracy of Six Classification Methods Where “rpart,” “ranger,” and “lr” Represent the Decision Tree, the Random Forest, and Logistic Regression, Respectively. Opens in a new window.Fig. 6. Balanced Accuracy of the Random Forest Running on the Original Dataset and Five Resampling Datasets. Opens in a new window.Fig. 7. Sensitivity of Six Classification Methods Where “rpart,” “ranger,” and “lr” Represent the Decision Tree, Random Forest, and Logistic Regression, Respectively. We observe that all classification methods have difficulty in classifying the positive observations. In particular, the KNN achieves the lowest sensitivity with only about 27%. Fig. 7 shows that with the help of the resampling methods, all classification methods are able to improve the true-positive rate with KNN being the one that most significantly improves (Table 7). Table 7. The Balanced Accuracy of Six Classification Models on the Original Dataset and Four Resampling Methods. Original Under Over SMOTE ROSE Decision tree 0.79 0.807 0.807 0.797 0.795 Random Forest 0.784 0.824 0.828 0.822 0.818 Logistic regression 0.794 0.819 0.82 0.812 0.818 KNN 0.581 0.616 0.608 0.62 0.587 SVM 0.787 0.82 0.826 0.821 0.821 AdaBoost 0.796 0.82 0.821 0.817 0.82 Fig. 5 also shows that all the resampling techniques help improve the overall performance of the classification models. While the decision tree, the Random Forest, the SVM, and AdaBoost give very similar good performances, KNN does not perform as well. Excluding SUR, the Random Forest is the best classification model. Combined with random over-sampling method, the Random Forest provides the best overall performance with the balanced accuracy of 0.8282. In addition, the average balanced accuracy of the Random Forest combined with four other resampling methods is also the highest: 0.8157. Of the six classification models, KNN gives the lowest average balanced accuracy (0.6028). Our method, SUR, does further improve the overall performance of the Random Forest, and it is better in doing so than the other four resampling techniques. Overall, SUR achieves the highest balanced accuracy, 0.837 (Table 1) Conclusion In this work, we study the performances of multiple classification methods on an imbalanced dataset with and without applying resampling techniques. Our computations have shown the effectiveness of applying resampling methods on imbalanced data before the modeling stage. All of the resampling techniques help improve the overall performances of the classification models. We also observe that the Random Forest works slightly better than other considered classification models. To further improve the performance of the Random Forest, we also propose an under-resampling procedure that uses the idea of eliminating “easier” observations before a traditional under-resampling method. Our proposed method works favorably compared to other combinations of classification and resampling methods. Although we use the Random Forest as a guide to classify “easy” observations, one can alternatively use other classifications to achieve this goal. This could be a direction for our future study. References Boser, Guyon, & Vapnik (1992) Boser, B. E. , Guyon, I. M. , & Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In D. Haussler (Ed.), COLT ’92: Proceedings of the Fifth Annual Workshop on Computational Learning Theory (pp. 144–152). New York, NY: ACM Press. Breiman, Friedman, Olshen, & Stone (1983) Breiman, L. , Friedman, J. H. , Olshen, R. A. , & Stone, C. J. (1983). Classification and regression trees. Belmont, TN: Wadsworth. Chawla, Bowyer, Hall, & Kegelmeyer (2002) Chawla, N. V. , Bowyer, K. W. , Hall, L. O. , & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357. Chen, Liaw, & Breiman (2004) Chen, C. , Liaw, A. , & Breiman, L. (2004). Using Random Forest to learn imbalanced data. Berkeley, CA: University of California. Cieslak & Chawla (2008) Cieslak, D. A. , & Chawla, N. V. (2008). Learning decision trees for unbalanced data. In W. Daelemans, B. Goethals, and K. Morik (Eds.), Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases – Part I, ECML PKDD’08 (pp. 241–256). Berlin: Springer-Verlag. Cortes & Vapnik (1995) Cortes, C. , & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. Cox (1958) Cox, D. R. (1958). The regression analysis of binary sequences (with discussion). Journal of the Royal Statistical Society B, 20, 215–242. Han, Wang, & Mao (2005) Han, H. , Wang, W. Y. , & Mao, B. H. (2005). Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. In Proceedings of the international conference on intelligent computing 2005 (pp. 878–887). Part I, LNCS 3644, 2005. Krawczyk (2016) Krawczyk, B. (2016). Learning from imbalanced data: Open challenges and future directions. Progress in Artificial Intelligence, 5(4), 221–232. Menardi & Torelli (2014) Menardi, G. , & Torelli, N. (2014). Training and assessing classification rules with imbalanced data. Data Mining and Knowledge Discovery, 28(1), 92122. Rubin (1986) Rubin, D. B. (1986). Statistical matching using file concatenation with adjusted weights and multiple imputations. Journal of Business & Economic Statistics, 4, 87–94. Vapnik & Lerner (1963) Vapnik, V. , & Lerner, A. (1963). Pattern recognition using generalized portrait method. Automation and Remote Control, 24, 774–780. Zikopoulos, Eaton, deRoos, Deutsch, & Lapis (2011) Zikopoulos, P. C. , Eaton, C. , deRoos, D. , Deutsch, T. , & Lapis, G. (2011). Understanding big data—analytics for enterprise class hadoop and streaming data (1st ed.). New York, NY: McGraw-Hill Osborne Media. Book Chapters Prelims Section A Marketing, Sales, and Service Forecasting Exploring the Suitability of Support Vector Regression and Radial Basis Function Approximation to Forecast Sales of Fortune 500 Companies Buy-online-and-pick-up-in-store Strategy and Showroom Strategy in the Omnichannel Retailing Service Contracts for Delays in Delivery Section B Economic, Financial, and Insurance Forecasting Growth, Business Cycles, and the Great Recession: Comparing State and County Unemployment Costs Per Capita for North Carolina Dimension Reduction in Bankruptcy Prediction: A Case Study of North American Companies Detecting Non-injured Passengers and Drivers in Car Accidents: A New Under-resampling Method for Imbalanced Classification Section C CEO Compensation and Operations Forecasting Regression Modeling of the Peer Group of Verizon Corporation for the CEO of Verizon Regression Modeling Based on a Peer Group for the Executive Compensation of AT&T CEO Agent-based Queuing Model for Call Center Forecasting and Management Optimization Index

Machine learning methods have recently gained attention in business applications. We will explore the suitability of machine learning methods, particularly support vector regression (SVR) and radial basis function (RBF) approximation, in forecasting company sales. We compare the one-stepahead forecast accuracy of these machine learning methods with traditional statistical forecasting techniques such as moving average (MA), exponential smoothing, and linear and quadratic trend regression on quarterly sales data of 43 Fortune 500 companies. Moreover, we implement an additive seasonal adjustment procedure on the quarterly sales data of 28 of the Fortune 500 companies whose time series exhibited seasonality, referred to as the seasonal group. Furthermore, we prove a mathematical property of this seasonal adjustment procedure that is useful in interpreting the resulting time series model. Our results show that the Gaussian form of a moving RBF model, with or without seasonal adjustment, is a promising method for forecasting company sales. In particular, the moving RBF-Gaussian model with seasonal adjustment yields generally better mean absolute percentage error (MAPE) values than the other methods on the sales data of 28 companies in the seasonal group. In addition, it is competitive with single exponential smoothing Advances in Business and Management Forecasting, Volume 13, 3 23 Copyright r 2019 by Emerald Publishing Limited All rights of reproduction in any form reserved ISSN: 1477-4070/doi:10.1108/S1477-407020190000013006 3 and better than the other methods on the sales data of the other 15 companies in the non-seasonal group.