Aligning Evaluation to Deployment: Measuring the Clinical Utility of End-to-End Deep Learning for Lung Cancer Screening

Background. Low-dose computed tomography (LDCT) screening reduces lung cancer mortality fundamentally through patient stratification, determining management intensity and screening intervals. Clinical guidelines such as Lung-RADS and NELSON make this assignment from nodule features, which can be measured by a radiologist or by a deep learning feature extractor. Emerging end-to-end deep learning approaches predict multi-year lung cancer (continuous) risk from a single LDCT, independent of nodule presence. These models are evaluated almost entirely by aggregate discrimination — AUC and the concordance index — which summarize ranking but do not localize where across the risk distribution a model is useful. They are not assessed as the actionable stratification tools they are meant to be. Methods. We propose an evaluation framework for end-to-end models that is aligned with clinical deployment and rooted in lung screening standards. It retains high-level ranking metrics (AUC, PR-AUC, concordance index) for comparability, and adds operating-point evaluation anchored to Lung-RADS: cancer capture from Lorenz curves, clinical utility from decision-curve analysis (DCA), calibration assessed both overall and within Lung-RADS categories. Because every operating point is a Lung-RADS operating point, the framework compares models with one another and with clinical standards such as Lung-RADS or NELSON. We applied it to three contemporary models that predict 6-year risk from a single LDCT on the National Lung Screening Trial (NLST) test cohort (6,223 scans / 2,199 patients), with retrospectively assigned Lung-RADS as the clinical standard and the non-imaging PLCOm2012 model as a baseline. Findings. The models have similar AUCs with overlapping confidence intervals (year-1 AUC 0.91–0.95). Calibration was good in aggregate but varied by category: the models over-predicted in the low-risk Lung-RADS tiers, were well calibrated at intermediate risk, and under-predicted in the high-risk tiers where workup is decided (11% predicted against 22% observed at the highest-risk tier). On cumulative-gains curves and decision-curve analysis, the stronger end-to-end models achieved at least on-par or higher cancer capture and greater net benefit than Lung-RADS, and the advantage was largest on baseline scans, where Lung-RADS has no prior CT for comparison, exceeding it by more than 25 percentage points in capture. Furthermore, the continuous scores were also effective at the extremes the categories cannot reach: they isolated very-high-risk patients at flag rates below the smallest Lung-RADS category, and recovered and ranked higher-risk patients placed in the benign Lung-RADS tiers, where the top score decile held about 70% of the cancers hidden in those tiers (6.7–7.4x enrichment). Implications. Evaluating end-to-end models as stratification tools, rather than by global discrimination, assesses them on the task they are designed for, and places them in the same frame as the clinical standards they would support: Lung-RADS, and feature-based deep learning models that classify by its rules. In that frame, end-to-end deep learning and Lung-RADS are not best understood as competitors. They derive risk differently and have complementary strengths, and our results indicate they are more useful combined than ranked against each other.