Enhanced Estimation of Local Intrinsic Dimensionality Using Auxiliary Distances

Estimating Intrinsic Dimensionality (ID) is of high interest in many machine learning tasks, including dimensionality reduction, outlier detection, similarity search and subspace clustering. Our proposed estimation strategy, ALID, makes use of a subset of the available intra-neighborhood distances to achieve faster convergence with fewer samples, and can thus be used on applications in which the data consists of many natural groups of small size. Moreover, it has a smaller bias and variance than state-of-the-art estimators, especially on nonlinear subspaces. We provide a theoretical analysis of the properties of the ALID estimator, and a thorough experimental framework that shows its faster convergence, smaller bias, and smaller variance compared with state-of-the-art estimators of ID.