Use of artificial intelligence and web scraping methods to retrieve information from the World Wide Web

The World Wide Web could represent a valid substitute for the traditional way of acquiring information. However, using this platform implies new challenges that are derived from the intrinsic nature of the information attained, which is expansive, sparse and mostly unstructured. We investigated the possibility of structuring data from different websites through web scraping techniques. Moreover, we exploited what is offered by some web search engines to progressively create queries that enabled us to select the most useful information. For this purpose we used a strategy that simulated the human behaviour to obtain timely price statistics will be included in the Agricultural Market Information System developed by FAO.