Chemical properties of lake water can provide valuable insight into its ecology. Lakes that are permanently frozen over with ice are generally inaccessible to comprehensive exploration by humans. This paper describes the integration of several novel and existing technologies into an autonomous underwater robot, ENDURANCE, that was successfully used for gathering scientific data in West Lake Bonney in Taylor Valley, Antarctica, in December 2008. This paper focuses on three novel technological and algorithmic solutions. First, a robust position estimation system that uses an acoustic beacon to complement traditional dead-reckoning is described. Second, a novel vision-based docking algorithm for locating and ascending a vertical shaft by tracking a blinking light source is presented. Third, a novel profiling system for measuring water properties while causing minimal water disturbance is described. Finally, experimental results from the scientific missions in 2008 in West Lake Bonney are presented.

The UT Austin Villa RoboCup 2003 Four-Legged Team was a new entry in the ongoing series of RoboCup legged league ompetitions. The team development began in mid-January of 2003, at whi h time none of the team members had any familiarity with the Aibos. Without using any RoboCup-related ode from other teams, we entered a team in the Ameri an Open ompetition at the end of April, and met with some su ess at the annual RoboCup ompetition that took pla e in Padova, Italy at the beginning of July. In this report, we des ribe both our development pro ess and the te hni al details of its end result, the UT Austin Villa team. The main ontributions of this paper are (i) a roadmap for new teams entering the ompetition who are starting from s rat h, and (ii) full do umentation of the algorithms behind our approa h with the goal of making them fully repli able.

This paper presents a vision-based docking algorithm for an autonomous underwater vehicle (AUV). The algorithm allows the AUV to egress through a melthole in the frozen surface of a lake after the AUV’s dead-reckoning system brings the vehicle in the vicinity of the melthole. A blinking light source is used to guide the robot towards the melthole and through it. A light detection and tracking algorithm performs a temporal analysis of images captured from an upward-facing camera to detect sources of illumination and identify and track the blinking target light source. The vehicle first moves in a spiral pattern to search for the target using the light-detection algorithm. On finding the light, the AUV ascends while keeping the light centered in the camera’s field of view. The vision-based docking algorithm was implemented on the ENDURANCE AUV and tested during a four-week-long scientific mission to explore West Lake Bonney in Antarctica in December 2008. The algorithm was used to ascend in 10 missions and to descend in 8 missions through a three-meter-deep melthole only slightly larger than the vehicle itself. In each instance, the vehicle was able to safely ascend or descend without coming into contact with the walls. Quantitative analysis of mission data confirmed that the tracking algorithm and ascent controller were robust and precise.

In order to claim fully general intelligence in an autonomous agent, the ability to learn is one of the most central capabilities. Classical machine learning techniques have had many significant empirical successes, but large real-world problems that are of interest to generally intelligent agents require learning much faster (with much less training experience) than is currently possible. This paper presents transfer learning, where knowledge from a learned task can be used to significantly speed up learning in a novel task, as the key to achieving the learning capabilities necessary for general intelligence. In addition to motivating the need for transfer learning in an intelligent agent, we introduce a novel method for selecting types of tasks to be used for transfer and empirically demonstrate that such a selection can lead to significant increases in training speed in a two-player game.

Recent work in transfer learning has succeeded in making reinforcement learning algorithms more efficient by incorporating knowledge from previous tasks. However, such methods typically must be provided either a full model of the tasks or an explicit relation mapping one task into the other. An autonomous agent may not have access to such high-level information, but would be able to analyze its experience to find similarities between tasks. In this paper we introduce Modeling Approximate State Transitions by Exploiting Regression (MASTER), a method for automatically learning a mapping from one task to another through an agent's experience. We empirically demonstrate that such learned relationships can significantly improve the speed of a reinforcement learning algorithm in a series of Mountain Car tasks. Additionally, we demonstrate that our method may also assist with the difficult problem of task selection for transfer.

A general game player is an agent capable of taking as input a description of a game's rules in a formal language and proceeding to play without any subsequent human input. To do well, an agent should learn from experience with past games and transfer the learned knowledge to new problems. We introduce a graph-based method for identifying previously encountered games and prove its robustness formally. We then describe how the same basic approach can be used to identify similar but non-identical games. We apply this technique to automate domain mapping for value function transfer and speed up reinforcement learning on variants of previously played games. Our approach is fully implemented with empirical results in the general game playing system.

A common goal for transfer learning research is to show that a learner can solve a source task and then leverage the learned knowledge to solve a target task faster than if it had learned the target task directly. A more difficult goal is to reduce the total training time so that learn- ing the source task and target task is faster than learning only the target task. This paper addresses the second goal by proposing a transfer hi- erarchy for 2-player games. Such a hierarchy orders games in terms of relative solution difficulty and can be used to select source tasks that are faster to learn than a given target task. We empirically test transfer between two types of tasks in the General Game Playing domain, the testbed for an international competition developed at Stanford. Our results show that transferring learned search heuristics from tasks in different parts of the hierarchy can significantly speed up search even when the source and target tasks differ along a number of important dimensions. task (called an auxiliary problem by Polya) is faster to solve than the target task, and the speedup in target task training time overcomes the time spent on learning the source task. To achieve this goal the learner must reason about all three steps. This paper takes a first step at the difficult problem of discovering appropriate source tasks by proposing a transfer hierarchy. Such a structure defines types of games that require more or less information to solve and thus may be used to order tasks by their relative solution complexity. Such an ordering can be used to identify source tasks that will take significantly less time to solve than a particular target task, re- ducing the impact of source task training on the total training time. In the future we hope that such a transfer hierarchy will be used to help automate the transfer learning process by assisting in the selec- tion of a source task for a given target task. In this paper we begin to evaluate the effectiveness of our proposed hierarchy by manu- ally constructing source tasks for a specified target task, where the selection of source task are motivated by the transfer hierarchy. To empirically demonstrate transfer between source and target task taken from our transfer hierarchy, we utilize the game of Mummy Maze. This game is an appropriate choice for two rea- sons. First, it has been released as a sample domain in the General Game Playing (Genesereth & Love 2005) (GGP) contest, an inter- national competition developed independently at Stanford. Second, the Mummy Maze task is easily modifiable so that it can conform to each task type in our transfer hierarchy. Our results show that a transferred heuristic is able to improve the speed of search by as much as 34%, meeting the target time goal, even if our source tasks differ from the target tasks along a number of dimensions. Addi- tionally, we demonstrate how the total training time goal may also be met for this particular pair of source and target types, depending on information gathering costs.

Keepaway soccer has been previously put forth as a testbed for machine learning. Although multiple researchers have used it successfully for machine learning experiments, doing so has required a good deal of domain expertise. This paper introduces a set of programs, tools, and resources designed to make the domain easily usable for experimentation without any prior knowledge of RoboCup or the Soccer Server. In addition, we report on new experiments in the Keepaway domain, along with performance results designed to be directly comparable with future experimental results. Combined, the new infrastructure and our concrete demonstration of its use in comparative experiments elevate the domain to a machine learning benchmark, suitable for use by researchers across the field.

Computer game players are typically designed to play a single game: today's best chess-playing programs cannot play checkers, or even tic-tac-toe. General Game Playing is the problem of designing an agent capable of playing many different previously unseen games. The first AAAI General Game Playing Competition was held at AAAI 2005 in order to promote research in this area. In this article, we survey some of the issues involved in creating a general game playing system and introduce our entry to that event. The main feature of our approach is a novel method for automatically constructing effective search heuristics based on the formal game description. Our agent is fully implemented and tested in a range of different games.

In a team-based multiagent system, the ability to construct a model of an opponent team's joint behavior can be useful for determining an agent's expected distribution over future world states, and thus can inform its planning of future actions. This paper presents an approach to team opponent modeling in the context of the RoboCup simulation coach competition. Specifically, it introduces an autonomous coach agent capable of analyzing past games of the current opponent, advising its own team how to play against this opponent, and identifying patterns or weaknesses on the part of the opponent. Our approach is fully implemented and tested within the RoboCup soccer server, and was the champion of the RoboCup 2005 simulation coach competition.

Mobile robots must cope with uncertainty from many sources along the path from interpreting raw sensor inputs to behavior selection to execution of the resulting primitive actions. This article identifies several such sources and introduces methods for (i) reducing uncertainty and (ii) making decisions in the face of uncertainty. We present a complete vision-based robotic system that includes several algorithms for learning models that are useful and necessary for planning, and then place particular emphasis on the planning and decision-making capabilities of the robot. Specifically, we present models for autonomous color calibration, autonomous sensor and actuator modeling, and an adaptation of particle filtering for improved localization on legged robots. These contributions enable effective planning under uncertainty for robots engaged in goal-oriented behavior within a dynamic, collaborative and adversarial environment. Each of our algorithms is fully implemented and tested on a commercial off-the-shelf vision-based quadruped robot.

RoboCup simulated soccer presents many challenges to reinforcement learning methods, including a large state space, hidden and uncertain state, multiple independent agents learning simultaneously, and long and variable delays in the effects of actions. We describe our application of episodic SMDP Sarsa(λ) with linear tile-coding function approximation and variable λ to learning higher-level decisions in a keepaway subtask of RoboCup soccer. In keepaway, one team, “the keepers,” tries to keep control of the ball for as long as possible despite the efforts of “the takers.” The keepers learn individually when to hold the ball and when to pass to a teammate. Our agents learned policies that significantly outperform a range of benchmark policies. We demonstrate the generality of our approach by applying it to a number of task variations including different field sizes and different numbers of players on each team.

The UT Austin Villa 2003 simulated online soccer coach was a first time entry in the RoboCup Coach Competition. In developing the coach, the main research focus was placed on treating advice-giving as a machine learning problem. Competing against a field of mostly hand-coded coaches, the UT Austin Villa coach earned first place in the competition. In this paper, we present the multi-faceted learning strategy that our coach used and examine which aspects contributed most to the coach’s success.

The UT Austin Villa Four-Legged Team for RoboCup 2006 was a fourth-time entry in the ongoing series of RoboCup legged league competitions. The team development began in mid-January of 2003 without any prior familiarity with the Aibos. After entering a fairly non-competitive team in RoboCup 2003, the team made several important advances. By the July 2004 competition that took place in Lisbon, Portugal, it was one of the top few teams. After those first two years of intense development, the team’s third and fourth years were devoted more to spin-off research than to development related to the competition. Building off of the team’s previous three technical reports [14, 15, 16], this report details the changes made to the team between RoboCup 2005 and RoboCup 2006 in Bremen. Taken together, this and the previous technical reports provide the history and details of the UT Austin Villa RoboCup Four-Legged team and the associated research in our lab.

Mobile robot localization, the ability of a robot to determine its global position and orientation, continues to be a major research focus in robotics. In most past cases, such localization has been studied on wheeled robots with range finding sensors such as sonar or lasers. In this paper, we consider the more challenging scenario of a legged robot localizing with a limited field-of-view camera as its primary sensory input. We begin with a baseline implementation adapted from the literature that provides a reasonable level of competence, but that exhibits some weaknesses in real-world tests. We propose a series of practical enhancements designed to improve the robot’s sensory and actuator models that enable our robots to achieve a 50% improvement in localization accuracy over the baseline implementation. We go on to demonstrate how the accuracy improvement is even more dramatic when the robot is subjected to large unmodeled movements. These enhancements are each individually straightforward, but together they provide a roadmap for avoiding potential pitfalls when implementing Monte Carlo Localization on vision-based and/or legged robots.

The UT Austin Villa 2003 simulated online soccer coach was a first time entry in the RoboCup Coach Competition. In developing the coach, the main research focus was placed on treating advice-giving as a learning problem. The coach learns to predict agent behavior from past observations and automatically generates advice to improve its team's performance. Using this approach, the UT Austin Villa coach earned first place in this year's competition.