IQAN-A distributed Actor Critic Approach to Reading Comprehension using Document Structure

During this project we set to take on the foundations laid down in Geva et al. (2018), which introduced a novel framework to navigate document structure for the reading comprehension task using deep reinforcement learning and improve upon it. The goal of the Reading Comprehension (RC) task is to enable machines to understand documents, and answer questions about them. Previous work on the RC task for long documents relied on scanning with a RNN entire documents tokenby-token which was time consuming even with a basic model. The Information Retrieval (IR) approach is to retrieve excerpts from the document and over these excerpts run an RNN, this approach is problematic since as document length increases, the efficiency of one-shot IR methods decreases and thousands of tokens are retrieved. In contrast, the work done in Geva et al. (2018) introduces a new approach, in the article documents were represented as trees, and a reinforcement learning agent learns to combine tokens from the document tree with feedback from a more expressive answer extraction model to navigate within the tree. The authors showed that it is possible to train an RL agent to successfully navigate a document while consuming a fraction of the tokens that a standard RC model uses. We follow up on this approach and further study the methods proposed. We accomplish this by redesigning of the code base used to allow use of any general actor-critic reinforcement learning framework for the TriviaQA-NoP dataset. We showcase our results using the IMPALA framework and the V-trace algorithm introduced in Espeholt et al. (2018), and denote our model IMPALA Question Answer Navigator (IQAN)2. In order to highlight the impact the new architecture has on training, we modify the original environment to use a sparse reward function, remove the replay buffer and do not employ tree sampling,