OTClean: Data Cleaning for Conditional Independence Violations using Optimal Transport
arxiv(2024)
摘要
Ensuring Conditional Independence (CI) constraints is pivotal for the
development of fair and trustworthy machine learning models. In this paper, we
introduce , a framework that harnesses optimal transport theory for data
repair under CI constraints. Optimal transport theory provides a rigorous
framework for measuring the discrepancy between probability distributions,
thereby ensuring control over data utility. We formulate the data repair
problem concerning CIs as a Quadratically Constrained Linear Program (QCLP) and
propose an alternating method for its solution. However, this approach faces
scalability issues due to the computational cost associated with computing
optimal transport distances, such as the Wasserstein distance. To overcome
these scalability challenges, we reframe our problem as a regularized
optimization problem, enabling us to develop an iterative algorithm inspired by
Sinkhorn's matrix scaling algorithm, which efficiently addresses
high-dimensional and large-scale data. Through extensive experiments, we
demonstrate the efficacy and efficiency of our proposed methods, showcasing
their practical utility in real-world data cleaning and preprocessing tasks.
Furthermore, we provide comparisons with traditional approaches, highlighting
the superiority of our techniques in terms of preserving data utility while
ensuring adherence to the desired CI constraints.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要