AskWorld: budget-sensitive query evaluation for knowledge-on-demand

Recently, several Web-scale knowledge harvesting systems have been built, each of which is competent at extracting information from certain types of data (e.g., unstructured text, structured tables on the web, etc.). In order to determine the response to a new query posed to such systems (e.g., is sugar a healthy food?), it is useful to integrate opinions from multiple systems. If a response is desired within a specific time budget (e.g., in less than 2 seconds), then maybe only a subset of these resources can be queried. In this paper, we address the problem of knowledge integration for on-demand time-budgeted query answering. We propose a new method, Ask World, which learns a policy that chooses which queries to send to which resources, by accommodating varying budget constraints that are available only at query (test) time. Through extensive experiments on real world datasets, we demonstrate Ask World's capability in selecting most informative resources to query within test-time constraints, resulting in improved performance compared to competitive baselines.