Advancing Software Project Effort Estimation: Leveraging a NIVIM for Enhanced Preprocessing

Software development effort estimation (SDEE) is essential for effective project planning and relies heavily on data quality affected by incomplete datasets. Missing data (MD) are a prevalent problem in machine learning, yet many models treat it arbitrarily despite its significance. Inadequate handling of MD may introduce bias into the induced knowledge. It can be challenging to choose optimal imputation approaches for software development projects. This article presents a novel incomplete value imputation model (NIVIM) that uses a variational autoencoder (VAE) for imputation and synthetic data. By combining contextual and resemblance components, our approach creates an SDEE dataset and improves the data quality using contextual imputation. The key feature of the proposed model is its applicability to a wide variety of datasets as a preprocessing unit. Comparative evaluations demonstrate that NIVIM outperforms existing models such as VAE, generative adversarial imputation network (GAIN), k$$ k $$-nearest neighbor (K-NN), and multivariate imputation by chained equations (MICE). Our proposed model NIVIM produces statistically substantial improvements on six benchmark datasets, that is, ISBSG, Albrecht, COCOMO81, Desharnais, NASA, and UCP, with an average improvement in RMSE of 11.05% to 17.72% and MAE of 9.62% to 21.96%.