Enhancing LLMs for Expert Question Answering by Synergizing with Knowledge Graphs

Recent developments have demonstrated the efficacy of enhancing the reliability and reasoning ability of Large Language Models (LLMs) through the integration of Knowledge Graphs (KGs). However, previous works have not fully considered the challenges present in expert domains, including the prevalence of professional terms with abbreviations and ambiguities, knowledge confusion stemming from the mixture of KGs with other retrieval methods, and others, which hinder their deployment in expert question-answering (QA) systems. In this work, we propose a QA framework for expert fields by Synergizing large Language models And knowledge Graphs (SLAG). SLAG is designed to improve the accuracy of KG retrieval and to intelligently coordinate KG retrievers with other retrieval methods through bidirectional enhancement between LLMs and KGs: (1) An LLM-Enhanced KG retriever is proposed to resolve entity ambiguities in queries and to accurately extract pertinent subgraphs. (2) A KG-Enhanced LLM reasoner is designed to distill subgraphs and to flexibly generate answers or explicit queries for cooperative retrievers. We compared SLAG with other retrieval techniques on multiple datasets across diverse expert domains, tasks, and languages. We observed that our approach achieved a relative improvement of 13% over state-of-the-art methods on public benchmarks in terms of the F1 score. Moreover, we successfully deployed SLAG in an industrial financial QA system and outperformed the latest online version. Finally, we provide an open source version of SLAG with code and datasets, which is available at https://anonymous.4open.science/r/SLAG-5611.