The underwater environment provides a range of interesting applications for human-robot teams. A critical issue for such teams is the development of an appropriate communication mechanism between humans and robots operating at depth. Humans operating at depth have developed an applied gesture-based communication language that can be leveraged to enable this communication, but it would be expensive and perhaps impractical to develop a hand-labelled dataset of these gestures to support a machine learning-based approach to the task. To avoid the cost of hand labelling such a large dataset, here we automate the process of collecting a labelled dataset through the use of a simple model trained on a hand-labelled dataset that only identifies salient objects (divers, their heads and hands), and then use a weakly supervised learning process to label a complex set of diver gestures. The result of this process is a system that can recognize a large number of diver hand gestures. Performance of the resulting system is compared against a hand-labelled set of diver gestures.

Invasive aquatic plant species, and in particular Eurasian Water-Milfoil (EWM), pose a major threat to domestic flora and fauna and can in turn negatively impact local economies. Numerous strategies have been developed to harvest and remove these plant species from the environment. However it is still an open question as to which method is best suited to removing a particular invasive species and the impact of different lake conditions on the choice. One problem common to all harvesting methods is the need to assess the location and degree of infestation on an ongoing manner. This is a difficult and error prone problem given that the plants grow underwater and significant infestation at depth may not be visible at the surface. Here we detail efforts to monitor EWM infestation and evaluate harvesting methods using an autonomous surface vessel (ASV). This novel ASV is based around a mono-hull design with two outriggers. Powered by a differential pair of underwater thrusters, the ASV is outfitted with RTK GPS for position estimation and a set of submerged environmental sensors that are used to capture imagery and depth information including the presence of material suspended in the water column. The ASV is capable of both autonomous and tele-operation.

Small and Medium-size Enterprises require increasingly versatile robots that are capable of learning new skills during their operating life span, in addition to the ability to integrate themselves into human teams as a new and effective partner. Teaching information and skills to a robot can quickly become very complex, especially when considering that the human partner of the robot is inexperienced in the field and does not have access to intuitive interaction channels to train such robots. In this work, a system is proposed that can virtually define an assembly task, based on CAD modelling that describe constraints between assembly parts. These constraints are then extracted in an ontology which is automatically translated into Simple Temporal Networks (STNs) leading to joint action plans.Results show the automatic translation between virtual teaching and knowledge acquired by the robot on a peg-in-hole problem before illustrating the connection to planning for human-robot collaborative tasks. As a proof of concept, these developments demonstrate that a CAD guided assembly planner can circumvent the need for skilled robot programming.

Robot-diver communication underwater is complicated by the attenuation of RF signals, the complexities of the environment in terms of deploying interaction devices, and issues related to the cognitive loading of human operators. Humans operating underwater have developed a simple yet effective strategy for diver-diver communication based on the visual recognition of gestures. Can a similar approach be effective for diver-robot communication? Here we present experiments with SCUBANet, an underwater detection dataset of body parts associated with diver-robot communication. Given the nature of standard diver gestures, here we concentrate on diver recognition and in particular on diver body-head-hand localization and examine the feasibility of using a CNN-based approach to address this problem. Such data-driven approaches typically require an appropriately annotated dataset. The SCUBANet dataset contains images of object classes commonly encountered during human-robot communication underwater. Object classes are labeled using per-instance bounding boxes. Annotations were created through crowd sourcing via a web-based interface to ease deployment. We provide baseline performance on diver and diver component recognition and localization using transfer learning on three widely available pre-trained models.

Current methods for human robot interaction in the underwater domain seem antiquated in comparison to their terrestrial counterparts. Visual tags and custom built wired remotes are commonplace underwater, but such approaches have numerous drawbacks. Here we describe a method for human robot interaction underwater that borrows from the long standing history of diver communication using hand signals; a three stage approach for diver-robot communication using a series of neural networks.

Communication with and control of underwater autonomous vehicles is complicated by the nature of the water medium which absorbs radio waves over short distances and which introduces severe limitations on the bandwidth of sound-based technologies. Given the limitations of acoustic and radio frequency (RF) communication underwater, light-based communication has also been used. Light-based communication is also emerging as an effective strategy for terrestrial communication. Can the emerging Light Fidelity (Li-Fi) communication standard be exploited underwater to enable devices in close proximity to communicate by light? This paper describes the development of the LightByte Li-Fi model for underwater use and experimental evaluation of its performance both terrestrially and underwater.

Teleoperation of unmanned underwater vehicles is most commonly facilitated through the use of expensive shielded ethernet cables or high-speed fibre optic cables that are also quite fragile. Wireless underwater communication has thus far has been dominated by bulky and expensive acoustic modems. Is it possible to exploit recent advances in visible light communication technology including Li-Fi as a replacement for these technologies? Here we describe a small scale Li-Fi system that can be used to provide short-range tele-operational control of an underwater vehicle. Such control can either be provided from a diver operating in close proximity of the robot or via a communications relay from surface-based support.

Simultaneous Localization and Mapping (SLAM) is a key stepping stone on the road to truly autonomous robots. SLAM is of particular importance to robots with large motion estimation problems, such as robots operating on the surface of aquatic GPS-denied environments where a paucity of local landmarks complicates SLAM and accurate navigation. Visual sensors have proven to be an effective tool for SLAM generally and have wide applicability, but is vision enough to solve SLAM in this environment, and how important are other sensors including a compass and water column depth to solve SLAM for an aquatic surface vehicle? Here we show that more sensors are almost always helpful in terms of improving SLAM performance in such a situation but that a compass is a particularly useful sensor for SLAM for autonomous surface vehicles; suggesting that a compass is a worthwhile investment for such a robot, and that compass alternatives should be considered when operating an autonomous vehicle in environments that are both GPS and compass-denied.

Although there are a large number of autonomous robot platforms for ground contact and flying robots, this has not been the case for underwater robotic platforms. This is not due to the lack of interesting applications in the shallow underwater domain (50m depth), but rather due to the relative cost of building such platforms. This has recently changed with the development of inexpensive thrusters and other underwater components. Leveraging these components and design principles learned from more expensive remotely operated vehicles this paper describes Milton, an inexpensive open hardware design for a traditional thruster-based underwater robot. Utilizing commercial off-the-shelf hardware and a ROS infrastructure, Milton, and Milton-inspired designs provide an inexpensive platform for autonomous underwater vehicle research.

This work couples the use of augmented and virtual reality, a tabletop display, and mobile devices (tablets and smartphones) to develop an innovative, system to support learner-centric anatomy education and training. The system provides a common tabletop interaction surface where a global view of an anatomical model is provided. This global view is available to all of the users (instructor and trainees) whom can interact with the model using the touch-sensitive tabletop display surface. In addition to this global view, each of the trainees has access to the model through a mobile device that is synchronized with the global view and provides each trainee with an individualized (local) view of the scene and interaction mechanisms. This paper outlines our integrated tabletop computer-tablet display and its use to facilitate virtual-based eye anatomy training.

Building a representation of space and estimating a robot's location within that space is a fundamental task in robotics known as simultaneous localization and mapping (SLAM). This work examines the problem of solving SLAM in aquatic environments using an unmanned surface vessel under conditions that restrict global knowledge of the robots pose. These conditions refer specifically to the absence of a global positioning system to estimate position, a poor vehicle motion model, and the lack of a strong stable magnetic field to estimate absolute heading. These conditions can be found in terrestrial environments where the line of sight to overhead satellites is occluded by surrounding structures and local magnetic inference disrupts reliable compass measurements. Similar conditions are anticipated in extra-terrestrial environments such as on Titan where the lack of a global satellite network inhibits the use of traditional positioning sensors and the lack of a stable magnetic core limits the applicability of a compass. This work develops a solution to the SLAM problem that utilizes shore features coupled with information about the depth of the water column. Theoretical results are validated experimentally using an autonomous surface vehicle utilizing omnidirectional video and a depth sounder. Solutions are compared to ground truth obtained using GPS.

The Sailing Stone project animates a stone sculpture such that a traditional, large scale, normally static installation physically interacts with the public. The Sailing Stone moves so as to disrupt the normal motion of visitors to the space so as to encourage changes in the nature of the interaction between visitors and the sculpture itself. The public gallery space is monitored through a network of video cameras, and this information is used to develop a model of human motion through the gallery space. Based on this model of visitor motion, a motion plan is developed and executed for the sculpture so as to maximally disrupt the motion paths of visitors.

The development of effective user interfaces for an autonomous system can be quite difficult, especially for devices that are to be operated in the field where access to standard computer platforms may be difficult or impossible. One approach in this type of environment is to utilize tablet or phone devices, which when coupled with an appropriate tool such as ROSBridge can be used to connect with standard robot middleware. This has proven to be a successful approach for devices with mature user interface requirements but may require significant software development for experimental systems. Here we describe RCON, a software tool that allows user interfaces on iOS devices to be configured on the device itself, in real time, in response to changes in the robot software infrastructure or the needs of the operator. The system is described in detail along with the accompanying communication framework and the process of building a user interface for a simple autonomous device.

Virtual reality systems are often proposed as an appropriate technology for the development of teleoperational interfaces for autonomous and semi-autonomous systems. In the past such systems have typically been developed as “one off” experimental systems in part due to a lack of common software systems for both robot software development and virtual environment infrastructure. More recently, common frameworks have begun to emerge for both robot control (e.g., ROS) and virtual environment display and interaction (e.g., Unity). Here we consider the task of developing systems that integrate these two environments. A yaml-based communications protocol over web sockets is used to glue the two software environments together. This allows each system to be controlled using standard software toolkits independently while providing a flexible interface between these two infrastructures.

Although many commercially available robots ship with a version of ROS this is not as true for many external sensors. There is a lack of ROS support for many devices and sensors one might use to extend the capabilities of a robot. As robots are deployed in more complex environments there is the need for more specialized sensors. In particular in the aquatic domain there is the need for support for depth sounders. This paper describes the design and construction process for building a ROS node for a NMEA 0183 compliant depth and temperature transducer and a strategy for extending this design to other NMEA devices.

Contactless motion sensing devices enable a new form of input that does not encumber the user with wearable tracking equipment. We present a novel travel technique using the Leap Motion finger tracker which adopts a 2DOF steering metaphor used in traditional mouse and keyboard navigation in many 3D computer games.

Robots with many degrees of freedom with one fixed end are know n astentacle robotsdue to their similarity to the tentacles found on squid and octopus. Tentacle robots ffer advantages over traditional robots in many scenarios due to their enhanced flexibility and reachabilit y. Planning practical paths for these devices is challenging due to their high degrees of freedom (DOFs). Samplin g-based path planners are a commonly used approach for high DOF planning problems but the solutions fo und using such planners are often not practical in that they do not take into account soft application-speci fic onstraints during the planning process. This paper describes a general sample adjustment method for tent acl robots, which adjusts the randomly generated nodes within their local neighborhood to satisfy soft c nstraints required by the problem. The approach is demonstrated on a planar tentacle robot composed of ten Ro botis Dynamixel AX-12 servos.

The underwater domain provides a wide range of potential applications for autonomous systems. Sessile (im-mobile) sensor platforms can provide a sensing network to monitor a range of different underwater events. Monitoring such networks can be a challenge, however, as the sensor nodes can be difficult to monitor and the nature of the medium limits wireless communication. Here we describe an approach that uses an autonomous underwater vehicle to monitor the state of sessile sensors. A visual communication channel is established from the sensor node to the robot that can then communicate the state of the sensor to an underwater-or surface-based operator. This paper describes the basic approach and results of preliminary experiments.