Deep Learning Based Hand-Drawn Molecular Structure Recognition and 3D Visualisation Using Augmented Reality

Due to both false-positive structure identification and flaws in the predicted structures, chemical structure identification from documents remains a complex challenge. Current techniques rely on customized rules and subroutines that, although usually effective, recognition rates are insufficient and systematic improvement is difficult at certain times. Especially when it comes to the identification of hand-drawn Lewis Structures of molecules, most of these existing methodologies fail. Therefore, through this research, we present a system to identify a camera-captured, hand-drawn Lewis structure of a molecule using Machine Learning and Deep Learning concepts. Due to less availability of hand-drawn Lewis structures, we had to make our own dataset and therefore the project was limited to 15 different hydro carbons. Moreover, we provide the users with a mobile application that can visualize the identified molecule in a 3-D space using Augmented Reality. Our machine learning model details are available on the Github (https://github.com/MZJGroup/MoleAR)