In the context of modern big data environments, traditional data quality (DQ) assessment methods (such as business rule-based methods, threshold-based validation, constraint checking, or DAMA) face significant limitations in terms of automation, adaptability and scalability. Also, the literature remains limited in terms of complete methodologies that guide the practical application of intelligent methods for DQ assessment. To face this limitations, this paper proposes a structured and replicable methodology for assessing data quality using machine learning (ML), addressing multiple DQ dimensions such as accuracy, completeness, timeliness, consistency, relevance, and credibility. The approach is built upon a modular pipeline that includes data collection from structured, semi-structured, and unstructured sources, enriched preprocessing phase with domain-aware rules, which refers to context-specific transformations and validations tailored to the semantics of the dataset. Model training uses various ML algorithms, and multi-metric evaluation aligned with DQ dimensions. A novel aspect of our work is the integration of progressive learning, with built-in versioning and traceability mechanisms that allow the system to track model evolution and compare different iterations over time, enabling incremental refinement of models. Experimental validation on five public datasets from diverse domains demonstrates the effectiveness of the proposed framework in enhancing data quality assessment precision, reducing manual effort, and improving model generalizability. This research sets the stage for a new generation of intelligent, self-improving data quality management systems adapted to dynamic and heterogeneous data ecosystems.
原文链接
分享
引用
