The Interdependence Between Cross-Validation and Hyperparameter Tuning Reveals Limits of Performance Metrics and Interpretation in Random-Forest PM₂.₅ Models

Abstract Random forest (RF) models for PM2.5 prediction inform exposure assessment and air-quality management. These models are typically evaluated using spatial or random cross-validation (CV), which often yields high R² values. However, how the choice of CV strategy interacts with hyperparameter tuning (e.g., max_features) has received little attention. Using daily PM2.5 data from 157 monitoring stations in northern China, we show that Spatial-CV and Temporal-CV produce substantially different performance (R² of 0.85 vs. 0.37) and, more importantly, lead to opposing optimal hyperparameter settings. The clearest case is max_features: Spatial-CV performs best when almost all features are considered at each split, while Temporal-CV performs best with only a few (20 vs. 3 of the 22 features). The performance gap and the hyperparameter reversal between Spatial-CV and Temporal-CV both reflect spatial interpolation rather than learned physical relationships. A diagnostic Date-ID model built from date and coordinate identifiers alone (no physical predictors) reproduced both patterns and reached the highest Spatial-CV R² (0.95); conversely, reducing station density from 157 to 20 stations progressively weakened both. For the current literature, the implication is direct: a high RF model score (e.g., R²) cannot certify what a model has learned or how far it generalizes. A predictor's assessed value and the model's overall interpretation depend on the data structure, evaluation, and hyperparameters, and can reverse when these change. Satellite aerosol optical depth (AOD), for instance, appears unnecessary under Spatial-CV but helps under Temporal-CV, and featureimportance rankings shift between the optimal hyperparameter configurations of Spatial-and Temporal-CV. This indicates a systematic mismatch between what a CV metric measures andwhat is claimed from it in the literature. We recommend tuning hyperparameters separately for each CV strategy, reporting performance across station densities, and building models using date-coordinate features as a baseline.