Visual Lifelog Retrieval: Humans and Machines Interpretation on First-Person Images

People usually forget the details of life experiences and encounter situations where they require to recall their past experiences. Therefore, lifelog retrieval turns out to be an emerging task in the AI community. Nowadays, people can record their life experiences by capturing images through wearable devices, writing blog posts, and so on. These personal big data stored in digital format can be considered lifelogs for retrieval. In this work, we focus on constructing a visual lifelog retrieval system that is able to efficiently find related images given the input textual queries. The core challenge of visual lifelog retrieval with textual queries comes from the semantic gap between visual and textual data. In this work, we propose LifeConcept, an interactive lifelog search system that is aimed at not only accelerating the retrieval process but also fetching more precise results. To reduce the semantic gap, we incorporate visual and textual concepts from images into our system utilizing pre-trained textual embeddings. Moreover, we propose a concept recommendation method enabling users to set up the related conditions for their requirements efficiently and search the desired images with appropriate query terms based on the suggestion. Experimental results show that textual concepts from images detected by CV models improve the retrieval results. We further employ annotators to label captions of images for investigating the difference between model-generated captions and human-labeled captions. The human-annotated dataset is released to facilitate future study of visual lifelog retrieval. Four research questions are discussed to explore the characteristic of models and humans interpreting the first-person images captured by wearable cameras. The impacts of model-generated captions and human-labeled captions in terms of visual lifelog retrieval are also included.