Amplification in Locality-Sensitive Hashing for Webshop Product Deduplication

The aggregation of highly heterogeneous product listings across online webshops suffers from severe scalability limitations and high false positive rates, as identifying duplicates without universal product identifiers typically requires computationally expensive pairwise comparisons. To resolve these specific challenges, this paper introduces two concrete enhancements to Locality-Sensitive Hashing (LSH) within the Multi-component Similarity Method with Pre-selection+ (MSMP+) that significantly improve both overall scalability and duplicate detection precision. First, we substitute the conventional MinHash sketching technique with Fast Similarity Sketching (FSS). FSS provides faster and more precise Jaccard similarity estimates, inherently reducing the time complexity of the initial hashing process. Second, we develop a parametrized amplification strategy leveraging iterative AND/OR operations on LSH functions to substantially minimize both false positive and false negative rates. Crucially, this amplification strategy improves scalability by allowing the algorithm to achieve higher accuracy while using fewer underlying hash functions, thereby directly decreasing the computational overhead of generating candidate pairs. We develop a comprehensive framework for these amplified LSH schemes, deriving optimal parameter configurations by minimizing a weighted error function that balances the trade-off between false positives and false negatives. Empirical evaluations, based on data collected from four webshops, including extensive bootstrap experiments, confirm that combining FSS with amplification consistently outperforms baseline, non-amplified MinHash configurations. To further validate the robustness and generalizability of our approach, we conducted an additional empirical analysis using the established Walmart-Amazon dataset. These supplementary experiments demonstrate that our optimized LSH framework maintains good deduplication precision across different, widely-used product domains. Ultimately, these targeted enhancements enable highly scalable duplicate detection, reflected in improved performance metrics such as pair completeness and pair quality, as well as an increased F1 score.