Enhancing Random Forest Algorithm Performance Using Non-Concave Penalization Techniques

In this paper, a novel hybrid framework, called the Sparse and Pruned Approach to Random Forest (SPARF), is proposed that enhances prediction accuracy by automatically pruning the ensemble of trees generated by the RF algorithm. Unlike traditional RF, the proposed framework applies non-concave penalties, namely SCAD and GSCAD, to identify and eliminate redundant trees. The core innovation lies in integrating SCAD-based techniques with RF and aggregating remaining trees. This model leverages the sparsity of SCAD and GSCAD to provide an interpretable model with high predictive accuracy. The performance of this method is evaluated on two real datasets and Monte Carlo simulation, where a combined RF, SCAD, and GSCAD model is created to reduce and automatically select RF trees. In real-world datasets, the SPARF-SC model achieves an approximately 9.68% reduction in RMSE compared to RF, while SPARF-GSC achieves a reduction of approximately 8.46% in RMSE.