An Inference-Based Approach to Recognizing Entailment

For this year's RTE challenge we have con- tinued to pursue a (somewhat) "logical" approach to recognizing entailment, in which our system, called BLUE (Boeing Language Understanding Engine) first cre- ates a logic-based representation of a text T and then performs simple inference (using WordNet and the DIRT inference rule da- tabase) to try and infer a hypothesis H. The overall system can be viewed as compris- ing of three main elements: parsing, WordNet, and DIRT, built on top of a sim- ple baseline of bag-of-words comparison. Ablation studies suggest that WordNet sub- stantially improves the accuracy scores, while, somewhat suprisingly, parsing and DIRT only marginally improve the accu- racy scores. We illustrate and discuss these results. Overall, BLUE's reasoning is sometimes insightful but sometimes non- sensical, the primary challenges being noise in the knowledge sources, lack of world knowledge, and the difficulty of ac- curate syntactic and semantic analysis. De- spite these challenges, we argue that form- ing semantic representations is a necessary first step towards the larger goal of ma- chine reading, and worthy of further explo- ration. Our best scores were 61.5% (2 way), 54.7% (3 way), and F=0.29 (Search Pilot).