Unsupervised Anomaly Thresholding from Reconstruction Errors.

Internet of Things (IoT) sensors generate massive streaming data which needs to be processed in real-time for many applications. Anomaly detection is one popular way to process such data and discover nuggets of information. Various machine learning techniques for anomaly detection rely on pre-labelled data which is very expensive and not feasible for streaming scenarios. Autoencoders have been found effective for unsupervised outlier removal because of their inherent ability to better reconstruct data with higher density. Our work aims to leverage this principle to investigate approaches through which the optimal threshold for anomaly detection can be obtained in an automated and adaptive fashion for streaming scenarios. Rather than experimentally setting an optimal threshold through trial and error, we obtain the threshold from the reconstruction errors of the training data. Inspired by image processing, we investigate how thresholds set by various statistical approaches can perform in an image dataset.