Automated Metadata Harmonization Using Entity Resolution and Contextual Embedding

Data curation process for Analytics and Data Science typically involves collecting data from large number of heterogenous and federated source systems with varied schema structures. To make these datasets interoperable, their metadata needs to be standardized. This process, also known as Metadata Harmonization, is predominantly a manual effort involving several hours of concentrated work that leads to reduced efficiency of ML-Ops lifecycle. This paper aims to demonstrate the automation of metadata harmonization using Machine Learning. It focuses on using entity resolution and contextual embedding methods to capture hidden relationships among data columns that help identify similarities in metadata, and thereby, help in automated mapping of columns to a standard schema. This study also addresses the automated derivation of the correct ontological structure for the target data model using ML. While prior competing approaches address manual metadata harmonization problem by proposing usage of semantic middleware, data dictionaries and matching rules this approach recommends novel usage of Machine Learning which improves efficacy of overall lifecycle.