A Deep Learning Framework for Memory Retrieval from Lifelogging Data

An emerging trend known as lifelogging is a process of digitally documenting and processing the data of an individual’s daily experiences. Lifelogging creates data which is continuous but can can be noisy; hence, it is challenging to find a comprehensive means of retrieving events or moments of interest to the public. This research proposes a deep learning framework to improve memory retrieval from lifelogging data. The proposed framework combines text-image embeddings and ensembles of a zero-shot deep learning model. The framework is implemented using three versions of the Contrastive Language-Image Pre-training (CLIP) model based on the combination of 12 datasets created by seven users containing more than 100,000 images. The results are evaluated based on the average precision@k metric for different values of k. Specifically, on the given dataset, the ensemble model consisting of ResNet50x64 and ViT-L/14 in the ratio 3:1 gives highest precision of 0.90 at k = 5. The proposed retrieval framework can be used to help people with Alzheimer’s and other forms of dementia for recalling useful information.