Behavior Oriented Design of Evaluation Function for Topic Crawler Search Algorithm

As the core of topic crawler search algorithm, the prevailing topic crawler search algorithm evaluation function cannot effectively deal with the local optimal problem in topic crawler search task. An evaluation function for crawler behavior and a corresponding memory-based calculation method are presented. Instead of evaluating the value of each object to be climbed, the method values the predictive results of the crawler's different behaviors. By querying the result of the similarity between the behavior and the behavior to be evaluated, the value of the behavior performed by the crawler under the current situation is obtained by weighting distance. Experimental results showed that the method have an improvement of more than 11.78% in harvest ratio and target recall, and more than 28.32% in velocity compared to baseline models.