Whether travelling, playing games, or debugging code, any situation where an agent desires change can be framed as a problem. Despite this ubiquity, there is no unifying framework describing how people reason backwards when solving problems. We introduce AND/OR trees, which chain together subgoals and actions to attain them, as a way to represent this process. To investigate whether actions from AND/OR trees were predictive of human behavior, we conducted a study in which participants solved deterministic, long-horizon puzzles. AND/OR trees were able to explain most of the actions the participants took. Next, we modeled search through these trees using a psychologically plausible, single-parameter search algorithm. We fit this model to the data of individual participants and found that it captures trends in summary statistics of human play. Our results show the promise of AND/OR trees as a representation for backward reasoning in problem solving.

A hallmark of human intelligence is the ability to plan multiple steps into the future(1,2). Despite decades of research(3-5), it is still debated whether skilled decision-makers plan more steps ahead than novices(6-8). Traditionally, the study of expertise in planning has used board games such as chess, but the complexity of these games poses a barrier to quantitative estimates of planning depth. Conversely, common planning tasks in cognitive science often have a lower complexity(9,10) and impose a ceiling for the depth to which any player can plan. Here we investigate expertise in a complex board game that offers ample opportunity for skilled players to plan deeply. We use model fitting methods to show that human behaviour can be captured using a computational cognitive model based on heuristic search. To validate this model, we predict human choices, response times and eye movements. We also perform a Turing test and a reconstruction experiment. Using the model, we find robust evidence for increased planning depth with expertise in both laboratory and large-scale mobile data. Experts memorize and reconstruct board features more accurately. Using complex tasks combined with precise behavioural modelling might expand our understanding of human planning and help to bridge the gap with progress in artificial intelligence.

In recent years, artificial intelligence has made great progress in improving machine performance in tasks that require planning many steps ahead. By comparison, cognitive science has lagged behind in understanding human planning in complex tasks. One question of long-standing interest in this domain is whether skilled decision-makers plan further into the future than novices. Traditionally, the study of expertise in planning has focused on board games like chess, but the complexity of these games poses a barrier to detailed behavioral modeling. Conversely, common planning tasks in cognitive science are often lower-complexity and impose a ceiling for the depth to which any player can plan. Here, we investigate expertise in a complex board game that offers ample opportunity for skilled players to plan deeply. Despite this complexity, we show that human behavior can be captured using a computational cognitive model based on heuristic search. To validate this model, we predict human choices, response times, eye movements and perform a Turing test. Using the model, we find robust evidence for increased planning depth with expertise in both laboratory and large-scale mobile data. Our results highlight the promise of investigating human planning in complex tasks with precise behavioral modeling.

People often choose between sticking with an available good option (exploitation) and trying out a new option that is uncertain but potentially more rewarding (exploration)1,2. Laboratory studies on explore-exploit decisions often contain real-world complexities such as non-stationary environments, stochasticity under exploitation and unknown reward distributions3-7. However, such factors might limit the researcher's ability to understand the essence of people's explore-exploit decisions. For this reason, we introduce a minimalistic task in which the optimal policy is to start off exploring and to switch to exploitation at most once in each sequence of decisions. The behaviour of 49 laboratory and 143 online participants deviated both qualitatively and quantitatively from the optimal policy, even when allowing for bias and decision noise. Instead, people seem to follow a suboptimal rule in which they switch from exploration to exploitation when the highest reward so far exceeds a certain threshold. Moreover, we show that this threshold decreases approximately linearly with the proportion of the sequence that remains, suggesting a temporal ratio law. Finally, we find evidence for 'sequence-level' variability that is shared across all decisions in the same sequence. Our results emphasize the importance of examining sequence-level strategies and their variability when studying sequential decision-making.

The original and corrected figures and equations are shown in the accompanying Publisher Correction.

How do people plan ahead in sequential decision-making tasks? In this article, we compare computational models of human behavior in a challenging variant of tic-tac-toe, to investigate the cognitive processes underlying sequential planning. We validate the most successful model by predicting choices during games, two-alternative forced choices and board evaluations. We then use this model to study individual skill differences, the effects of time pressure and the nature of expertise. Our findings suggest that people perform less tree search under time pressure, and that players search more as they improve during learning.

Human cognition inspired the earliest algorithms for game-playing computer programs. However, the studies of human and computer game play quickly diverged: the Artificial Intelligence community focused on theory and techniques to solve games, while behavioral scientists empirically examined simple decision-making in humans. In this paper, we combine concepts and methods from the two fields to investigate whether human and AI players take similar approaches in an adversarial combinatorial game. We develop and compare five models that capture human behavior. We then demonstrate that our models can predict behavior in two related tasks. To conclude, we use our models to describe what makes a strong human player.

Monte-Carlo Tree Search (MCTS) algorithms estimate the value of MDP states based on rewards received by performing multiple random simulations. MCTS algorithms can use different strategies to aggregate these rewards and provide an estimation for the states’ values. The most common aggregation method is to store the mean reward of all simulations. Another common approach stores the best observed reward from each state. Both of these methods have complementary benefits and drawbacks. In this paper, we show that both of these methods are biased estimators for the real expected value of MDP states. We propose an hybrid approach that uses the best reward for states with low noise, and otherwise uses the mean. Experimental results on the Sailing MDP domain show that our method has a considerable advantage when the rewards are drawn from a noisy distribution.

In the Canadian Traveler Problem (CTP) a traveling agent is given a graph where some of the edges may be blocked with a known probability. The agent has to travel from a given start state to a given goal state. A solution for CTP is a policy, that has the smallest expected traversal cost. CTP is kn own to be intractable. Previous work has focused on the task of performing a single trip. We generalize CTP to its repeated task version where a number of trips from the start to the goal should be performed (either by the same agent or by multiple agents) while the aim is to minimize the total travel cost in all trips. We provide optimal algorithms for the special case of repeated-task CTP on disjoint path graphs. Based on these findings, we provide a scheme for solving the problem on general graphs. According to this scheme, we first solve a simplified variant of the problem and then apply the solution to the original problem. Empirical results show the benefits of the suggested scheme. For small graphs, where we could compare to optimal policies, our approach achieves near-optimal results with only a fraction of the computation cost. We also provide suboptimal solutions based on UCT that use our heuristics scheme. Experimental results show the benefits of using our scheme for UCT.

In the Multi-Agent Path Finding problem (MAPF), we are given a map and a set of agents with distinct source and goal locations. The task is to compute a path for each agent from its initial location to its goal location without conflicting with other agents.MAPF solvers can be divided into classes based on their purpose. One of these classes is the class of online MAPF algorithms, in which the search for paths is interleaved with the actual physical moves of the agents. A prominent algorithm in this class is the Windowed Hierarchical Cooperative A* algorithm (WHCA*) where paths are planned for each agent individually and cooperation is obtained using a reservation table.A number of extensions for WHCA* already exist. In this paper we propose a general approach for the baseline WHCA* algorithm which is orthogonal to all other existing extensions. We improve WHCA* by introducing the Conflict Oriented (CO) principle for focusing the agent coordination around conflicts. In addition, we provide a conflict-oriented prioritization mechanism that intelligently chooses which agent should act next. Experimental results demonstrate the advantage of our approach over WHCA*.

We introduce a new variant of the multi-armed bandit problem, called Volatile Multi-Arm Bandit (VMAB). A general policy for VMAB is given with proven regret bounds. The problem of collecting intelligence on profiles in social networks is then modeled as a VMAB and experimental results show the superiority of our proposed policy.

In many cases the best way to find a profile or a set of profiles matching some criteria in a social network is via targeted crawling. An important challenge in targeted crawling is to choose the next profile to explore. Existing heuristics for targeted crawling are usually tailored for specific search criterion and could lead to short-sighted crawling decisions. In this paper we propose and evaluate a generic approach for guiding a social network crawler that aims to provide a proper balance between exploration and exploitation based on the recently introduced variant of the Multi-Armed Bandit problem with volatile arms (VMAB). Our approach is general-purpose. In addition, it provides provable performance guarantees. Experimental results indicate that our approach compares favorably with the best existing heuristics on two different domains.

Multi-agent pathfinding (MAPF) deals with planning paths for individual agents such that a global cost function (e.g., the sum of costs) is minimized while avoiding collisions between agents. Previous work proposed centralized or fully cooperative decentralized algorithms assuming that agents will follow paths assigned to them. When agents are {\em self-interested}, however, they are expected to follow a path only if they consider that path to be their most beneficial option. In this paper we propose the use of a taxation scheme to implicitly coordinate self-interested agents in MAPF. We propose several taxation schemes and compare them experimentally. We show that intelligent taxation schemes can result in a lower total cost than the non coordinated scheme even if we take into consideration both travel cost and the taxes paid by agents.

In many cases the best way to find a profile or a set of profiles matching some criteria in a social network is via targeted crawling. An important challenge in targeted crawling is choosing the next profile to explore. Existing heuristics for targeted crawling are usually tailored for specific search criterion and could lead to short-sighted crawling decisions. In this paper we propose and evaluate a generic approach for guiding targeted crawling which is based on recent developments in Artificial Intelligence. Our approach, based on the recently introduced variant of the Multi-Armed Bandit problem with volatile arms (VMAB), aims to provide a proper balance between exploration and exploitation during the crawling process. Unlike other heuristics which are hand tailored for specific type of search queries, our approach is general-purpose. In addition, it provides provable performance guarantees. Experimental results indicate that our approach compares favorably with the best existing heuristics on two different domains.

In this paper we introduce the Target Oriented Network Intelligence Collection (TONIC) problem, which is the problem of finding profiles in a social network that contain information about a given target via automated crawling. We formalize TONIC as a search problem and a best-first approach is proposed for solving it.Several heuristics are presented to guide this search.These heuristics are based on the topology of the currently known part of the social network.The efficiency of the proposed heuristics and the effect of the graph topology on their performance is experimentally evaluated on the Google+ social network.

In the Canadian Traveler Problem (CTP) a traveling agent is given a graph, where some of the edges may be blocked, with a known probability. A solution for CTP is a policy, that has the smallest expected traversal cost. CTP is intracable. Previous work has focused on the case of a single agent. We generalize CTP to a repeated task version where a number of agents need to travel to the same goal, minimizing their combined travel cost. We provide optimal algorithms for the special case of disjoint path graphs. Based on a previous UCT-based approach for the single agent case, a framework is developed for the multi-agent case and four variants are given two of which are based on the results for disjoint-path graphs. Empirical results show the benefits of the suggested framework and the resulting heuristics. For small graphs where we could compare to optimal policies, our approach achieves near-optimal results at only a fraction of the computation cost. Introduction The Canadian Traveler Problem (CTP) (Nikolova and Karger 2008; Papadimitriou and Yannakakis 1991; Bar-Noy and Schieber 1991) is a navigation problem where traveling agent receives a graph as input and needs to travel from its initial location to a given goal location. The complication in CTP is that some edges of the graph may be blocked with a known probability. The basic action in the CTP is a move action along an edge of the graph. Moves incur costs. After arriving at a node, the agent can sense its incident edges at no cost. The task is to minimize the travel cost of reaching the goal. Since some of the graph’s edges may be blocked, a simple search for a path does not work; a solution is a contingent plan, or policy, that has the smallest expected traversal cost. In the basic version of the CTP, a single agent is assumed. However, in many realistic settings the problem needs to be solved for a group of agents, requiring minimization of the combined travel cost of all agents. For example, think of an owner of a fleet of trucks, who has to move the trucks from a single source to a single destination. In this paper, we generalize CTP to its multi-agent repeated-task setting, that assumes that the agents traverse the graph sequentially. We also briefly examine the interleaved version, where agents Copyright c © 2011, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. are allowed to traverse the graph concurrently. Single-agent CTP is intractable (Papadimitriou and Yannakakis 1991). Having multiple agents further increases the size of the statespace, which makes finding an optimal solution even harder and is also intractable. Therefore, we cannot find optimal solutions efficiently in general. However, we provide efficient optimal solutions for special case graphs which are then used to generate heuristics for the general case. These heuristics are tested empirically against baselines developed for single and multi-agent. Experimental results show that our heuristics appear to improve the expected travel cost, and are not too far from the optimal for small graphs. Variants of the Canadian Traveler Problem In the single agent version of CTP, a traveling agent is given a connected weighted graph G = (V,E), a source vertex (s ∈ V ), and a target vertex (t ∈ V ). The input graph G may undergo changes, that are not known to the agent, before the agent begins to act, but remains fixed subsequently. In particular, some of the edges in E may become blocked and thus untraversable. Each edge e ∈ E has a weight, w(e), and is blocked with a known blocking probability p(e), or traversable with probability q(e) = 1− p(e). The agent can perform move actions along an unblocked edge which incurs a travel cost equal to w(e). The status of an edge (blocked or traversable) is revealed to the agent, only after the agent reaches a vertex incident to that edge. The task of the agent is to travel from s to t with minimal total travel cost. As the exact travel cost is uncertain until the end, a solution to CTP is a traveling strategy (policy) which yields a small (ideally optimal) expected travel cost. To illustrate CTP, consider figure 1 with unknown edges e0,1 and e1,1. If e0,1 is traversable, the cheapest path is (s, v0, t) with cost 1.5, but if e0,1 is blocked, and e1,1 is traversable, the cheapest path is (s, v1, t) with cost 2.5. Taking into account the blocking events (e0,1 is likely to be blocked, and its status can only be observed from v0), the optimal policy is to try v1 first, then if e1,1 is blocked go to v0; finally if e0,1 is also blocked, reach t though path I2.

The Canadian Traveler Problem (CTP) is a navigation problem where a graph is initially known, but some edges may be blocked with a known probability. The task is to minimize travel effort of reaching the goal. We generalize CTP to allow for remote sensing actions, now requiring minimization of the sum of the travel cost and the remote sensing cost. Finding optimal policies for both versions is intractable. We provide optimal solutions for special case graphs. We then develop a framework that utilizes heuristics to determine when and where to sense the environment in order to minimize total costs. Several such heuristics, based on the expected total cost are introduced. Empirical evaluations show the benefits of our heuristics and support some of the theoretical results.