The collaboration between humans and robots is one of the most disruptive and challenging research areas. Even considering advances in design and artificial intelligence, humans and robots could soon ally to perform together a number of different tasks. Robots could also became new playmates. In fact, an emerging trend is associated with the so-called phygital gaming, which builds upon the idea of merging the physical world with a virtual one in order to let physical and virtual entities, such as players, robots, animated characters and other game objects interact seamlessly as if they were all part of the same reality. This paper specifically focuses on mixed reality gaming environments that can be created by using floor projection, and tackles the issue of enabling accurate and robust tracking of off-the-shelf robots endowed with limited sensing capabilities. The proposed solution is implemented by fusing visual tracking data gathered via a fixed camera in a smart environment with odometry data obtained from robot's on-board sensors. The solution has been tested within a phygital gaming platform in a real usage scenario, by experimenting with a robotic game that exhibits many challenging situations which would be hard to manage using conventional tracking techniques.

Commercial off-the-shelf (COTS) robots are becoming ever more ubiquitous. Among the most common applications there are toy robots. In order to keep their cost down, these robots are usually equipped with the minimum set of sensors necessary for their basic functioning. Specifically, to add entertainment value, they often feature a video camera and, in most of the games developed so far, players are basically tele-operating them through a smartphone app. The present paper aims to show how to provide existing consumer-grade robots with new capabilities to transform them in more appealing gaming companions. In particular, by considering as a test bench a wheeled, non-holonomic COTS robot whose only accessible sensor is a low-resolution camera, previously unavailable localization and autonomous navigation capabilities are developed for it by exploiting dead reckoning and artificial landmark detection algorithms. These capabilities could then be exploited to create new types of games, which can be played in free-scale unknown environments and feature new forms of interaction. Implementation details of a sorting game in which the robot acts as a referee are reported.

This paper aims to investigate the impact associated with the introduction of emotional features and autonomous behaviors in robotic games leveraging drones. To this purpose, a game named Protoman Revenge incorporating the above elements is implemented by following known guidelines regarding game design, in general, and so-called Physically Interactive Robotic Games (PIRGs), in particular. A user study is devised to evaluate how the above traits affect the individual's experience and engagement. Obtained results could be possibly exploited to orient further research in the field of robotic gaming and other applications (not necessarily involving drones) w.r.t. to the perception of robot's autonomy and of emotional encoding strategies.

In this article we present the Phygital Game project, a mixed-reality game platform in which children can play with or against a robot. The project was developed by adopting a human-centered design approach, characterized by the engagement of both children and parents in the design process, and situating the game platform in a real context—an educational center for children. We report the results of both the preliminary studies and the final testing session, which focused on the evaluation of usability factors. By providing a detailed description of the process and the results, this work aims at sharing the findings and the lessons learned about both the implications of adopting a human-centered approach across the whole design process and the specific challenges of developing a mixed-reality playground.

Due to the continuous advancements made in robot technologies, the development of intuitive and effective user interfaces for human-robot interaction is getting increasingly important. This paper investigates how different types of interfaces can be used for allowing a single operator to remotely control a team of robots endowed with different capabilities. Attention is focused on three user interfaces based on a gamepad, on a mobile device and on hand tracking, respectively. To evaluate pros and cons of the above interfaces, a user study was conducted, in which participants had to combine the capabilities of a rover, a drone and a robotic arm in order to carry out a search and pick task. Based on the experiments, the fastest way to complete the task was to use the mobile device. However, results showed that some of the interfaces could provide better performances for selected robots and associated sub-tasks. It is worth observing that, despite evidences about efficiency, participants rated the gamepad as the preferred interface from the point of view of subjective usability.

Augmented Reality (AR) is expected to change the way we play, by transforming the world around us in an incredibly rich gaming environment. In this work, connected robots and natural interaction means are combined with projected AR to create a gaming experience more physical and engaging.

This paper presents a framework for automatically generating speech-based interfaces for controlling virtual and augmented reality (AR) applications on wearable devices. Starting from a set of natural language descriptions of application functionalities and a catalog of general-purpose icons, annotated with possible implied meanings, the framework creates both vocabulary and grammar for the speech recognizer, as well as a graphic interface for the target application, where icons are expected to be capable of evoking available commands. To minimize user's cognitive load during interaction, a semantics-based optimization mechanism was used to find the best mapping between icons and functionalities and to expand the set of valid commands. The framework was evaluated by using it with see-through glasses for AR-based maintenance and repair operations. A set of experimental tests were designed to objectively and subjectively assess first-time user experience of the automatically generated interface in relation to that of a fully personalized interface. Moreover, intuitiveness of the automatically generated interface was studied by analyzing the results obtained through trained users on the same interface. Objective measurements (in terms of false positives, false negatives, task completion rate, and average number of attempts for activating functionalities) and subjective measurements (about system response accuracy, likeability, cognitive demand, annoyance, habitability, and speed) reveal that the results obtained by the first-time users and experienced users with the proposed framework's interface are very similar, and their performances are comparable with those of both the considered references.

Augmented Reality (AR) has been proved to be an effective tool to improve and enhance the learning experience of students. On the other hand, issues regarding the inflexibility of AR contents can strongly limit the usability of AR applications in education. This paper presents results obtained by using the AR framework designed and developed for the EASE-R $$^{3}$$ European project and focused on the generation of maintenance procedures for machine tools. The high system flexibility allows instructors to easily make maintenance procedures suitable for the skill level of technicians to be trained. A case study is presented and results gathered so far analyzed and assessed.

Augmented Reality (AR) applications are nowadays largely diffused in many fields of use, especially for entertainment, and the market of AR applications for mobile devices grows faster and faster. Moreover, new and innovative hardware for human-computer interaction has been deployed, such as the Leap Motion Controller. This paper presents some preliminary results in the design and development of a hybrid interface for hand-free augmented reality applications. The paper introduces a framework to interact with AR applications through a speech and gesture recognition-based interface. A Leap Motion Controller is mounted on top of AR glasses and a speech recognition module completes the system. Results have shown that, using the speech or the gesture recognition modules singularly, the robustness of the user interface is strongly dependent on environmental conditions. On the other hand, a combined usage of both modules can provide a more robust input.

This paper addresses the issue of screen time, a growing phenomenon triggered by the spread of digital devices, which may cause a health impact. The reduction of screen time emerges as a requirement, especially in games. For this reason, the paper offers an overview of the current Phygital game trend, through some related projects that highlight the potential of physical features in virtual games in terms of attraction and engagement. A new gaming framework is proposed as a step forward in the virtual-physical contamination. The proposed project consists of a projected playground, on which people and robot(s) can physically interact. The role of the robot is enhanced entrusting it different roles, from a companion, to an adversary, to the avatar of a remote player. By abstracting the inherent complexities and providing common functionality, the proposed framework aims to simplify the development of games that wish to exploit both digital and physical elements.