The complexity of humanoid robots necessitates the development of robust algorithms. This paper introduces a Dual-Actor Proximal Policy Optimization (DA-PPO) algorithm, a novel reinforcement learning approach designed to enhance humanoid robot navigation and movement capabilities in complex environments. By implementing a dual-actor system, DA-PPO shows a significant improvement in walking over obstacles and navigation tasks com pared to state-of-the-art reinforcement learning algorithms such as Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Soft Actor Critic (SAC), and Twin Delayed DDPG (TD3). Our experimental results, conducted in the Isaac Gym environment, demonstrate that our proposed DA-PPO excels in both flat and challenging terrain navigation in different walking scenarios. DA-PPO maintains an improvement of 77% over PPO on average and more than doubles the reward achieved by DDPG, TD3, and SAC. This research demon strates the potential of DA-PPO in robotic applications, utilizing a popular existing humanoid robot platform, and highlights its adaptability and efficiency in controlling the complex walking of humanoid robots.

In spite of the ease with which humans can pick up and use objects, object manipulation remains a popular and challenging research topic even after many decades. The object’s shape, weight, or position can make it impossible for the robot to manipulate it with a single arm. In such situations, dual-arm manipulators are needed. Furthermore, manipulation may require dragging or pushing objects. Dragging an object while considering the desired orientation of the object, along with uncertainties like friction on the surface and possible slipping of the object in the robot’s hands, is very challenging. Therefore, in this paper, we introduce a novel hierarchical deep deterministic policy gradient (HDDPG) that exploits the continuity of the state and action spaces based on the actor-critic, model-free algorithm as a strategy controller to solve the dual-arm object dragging problem. To evaluate the proposed algorithm, we conduct extensive experiments both in simulation and on a real adult-sized humanoid robot. We use 13 different objects, including keyboards, laptops, boxes, etc. These experiments demonstrate the effectiveness and high performance of the proposed algorithm, with an average success rate of 97.3

Entertainment robotics has garnered significant attention in recent years, with researchers focusing on developing robots capable of performing a variety of tasks, including magic, drawing, dancing, and music. This article presents our research on forming a musical band that includes both humanoid robots and human musicians, with the goal of achieving natural synchronization and collaboration during musical performances. We utilized two of our humanoid robots for this project: Polaris, a mid-sized humanoid robot, as the drummer, and Oscar, a Robotis-OP3 humanoid robot, as the keyboardist. The technical implementation incorporated essential components such as visual servoing, human-robot interaction, and Robot Operating System (ROS), enabling seamless communication and coordination between the humanoid robots and the human musicians. The success of this collaborative effort can be both seen and heard through the following YouTube link: https://youtu.be/pFOyt1KKCfY?feature=shared.

This paper introduces a novel multi-agent Deep Reinforcement Learning (DRL) framework named the Cooperative Dual-Actor Proximal Policy Optimization (CDA-PPO) algorithm, designed to address complex humanoid robot cooperative learning control tasks. Effective cooperation among multiple humanoid robots, particularly in scenarios involving complex walking gait control and external disturbances in dynamic environments, is a critical challenge. This is especially pertinent for tasks requiring precise coordination and control, such as joint object transportation. In various real-life scenarios, humanoid robots might need to cooperate to carry large objects in many scenarios. This capability is crucial for logistics, manufacturing, intelligent transportation, and search-and-rescue missions applications. Humanoid robots have gained significant popularity, and their use in these cooperative tasks is becoming more common. To address this challenge, we propose CDA-PPO, which introduces a learning-based communication platform between agents and employs two distinct policy networks for each agent. This dual-policy approach enhances the robots’ ability to adapt to complex interactions and maintain stability while performing intricate tasks. We demonstrate the efficacy of CDA-PPO in a cooperative object-transportation scenario, where two humanoid robots collaborate to carry a table. The experimental results show that CDA-PPO significantly outperforms traditional methods, such as Independent PPO (IPPO), Multi-Agent PPO (MAPPO), and Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), in terms of training efficiency, stability, reward acquisition, and humanoid robot cooperative balance control with effective coordination between robots. The findings underscore the potential of CDA-PPO to advance the field of cooperative multi-agent control problems, proposing the way for future research in complex robotics applications.

Robotic arms are increasingly popular in a variety of applications, and new designs still need to be explored. In this paper, we introduce our newly designed 6-axis next generation pioneer arm robot named A6-NXP to solve an industrial problem. Our robot, in contrast to existing robotics platforms, is designed with high-performance BLDC (Brushless DC) motors with planetary gear reducers to achieve high torque density and quick, precise motion. This gives our robot the highly dynamic motion control ability beside the precise control task performance. In this study to show the performance of our robot design we solve Inverse Kinematics (IK) and trajectory planning, beside the monocular vision to detect objects orientation. The robot has been successfully applied in industrial scenarios such as object manipulation and drill sharpening showing performance of our robotics arm.

Intelligent robotics bridges the gap between the digital and physical worlds, which requires a solid understanding of many different aspects. In this paper, we describe our approach to a gentle introduction into robotics control in general and control theory in particular using a small electric airplane with an Arduino micro-controller. The microcontroller has a built-in pressure sensor, which allows surprisingly efficient and accurate control of the altitude of the plane. In addition to concepts and techniques in aerodynamics, flight control is an extremely motivating domain to introduce students to the necessity for and approaches to open-loop and feedback control. We tested the approach with great success with a group of about 30 students who participated in the Frontiers Camp of the FIRA Asia Cup 2024.

Legged robots have been a prominent focus of research for an extensive period, owing to their enhanced stability and maneuverability in challenging terrains compared to wheeled robots. In recent decades, the application of reinforcement learning (RL) to train legged robots has yielded excellent results. This approach has effectively addressed numerous challenges that traditional methods struggled to overcome, such as navigating through complex environments. The advancements in RL for legged robots have significantly improved the feasibility and success of demanding applications, including exploration and resource delivery in outdoor settings. This paper introduces a training architecture for a hexapod robot based on the implementation of proximal policy optimization (PPO) on a GPU. This architecture enables the hexapod to transport objects across uneven terrain surfaces while adhering to input control commands. Our investigation extends to assessing the performance limitations of the hexapod robot in both flat and uneven ground environments, with a particular focus on evaluating the impact of different activation functions.

Abstract Recently, the field of robotics development and control has been advancing rapidly. Even though humans effortlessly manipulate everyday objects, enabling robots to interact with human-made objects in real-world environments remains a challenge despite years of dedicated research. For example, typing on a keyboard requires adapting to various external conditions, such as the size and position of the keyboard, and demands high accuracy from a robot to be able to use it properly. This paper introduces a novel hierarchical reinforcement learning algorithm based on the Deep Deterministic Policy Gradient (DDPG) algorithm to address the dual-arm robot typing problem. In this regard, the proposed algorithm employs a Convolutional Auto-Encoder (CAE) to deal with the associated complexities of continuous state and action spaces at the first stage, and then a DDPG algorithm serves as a strategy controller for the typing problem. Using a dual-arm humanoid robot, we have extensively evaluated our proposed algorithm in simulation and real-world experiments. The results showcase the high efficiency of our approach, boasting an average success rate of 96.14% in simulations and 92.2% in real-world settings. Furthermore, we demonstrate that our proposed algorithm outperforms DDPG and Deep Q-Learning, two frequently employed algorithms in robotic applications.

In recent years, there has been a growing interest in entertainment robotics, with researchers exploring the creation of robots capable of performing a wide range of tasks, such as magic, art drawing, dancing, and music. This paper highlights our research in creating a musical band with humanoid robots and human musicians, with the goal of achieving natural synchronization and collaboration in musical performances. We selected two of our humanoid robots namely Polaris, a mid-sized humanoid robot, serving as the drummer, and Oscar, a Robotis-OP3 humanoid robot, serving as the keyboardist. The technical implementation included key elements like visual servoing, humanrobot interaction, and ROS, which facilitated seamless communication and coordination between our humanoid robots and human musicians. The successful collaboration of our robots with human musicians can be visually and audibly appreciated through the following YouTube link: https://youtu.be/pFOyt1KKCfY.

This study explores the performance of Convolutional Neural Networks (CNNs) in the context of humanoid robot localization in dynamic environments. Utilizing a front-mounted camera system, initial experiments demonstrate CNNs achieving a 72% accuracy in position and a 92% accuracy rate in orientation with an 8000-image dataset. These results underscore the effectiveness of CNNs in addressing the challenge of precise robot localization. Moreover, the study introduces the YOLO (You Only Look Once) object detection algorithm to further enhance performance. Beyond robotics, this research extends to applications in smartphone navigation, Indoor GPS systems, and drone tracking. The paper provides insights into the methodologies employed and highlights the transformative potential of integrating CNNs into localization tasks.

This paper presents the technical specifications of a lightweight humanoid robot platform named Robinion Sr. including its mechanical and electrical design. We describe a versatile and robust mechatronic system, efficient walking gait, and software architecture of the humanoid robot. The humanoid robot platform is targeted for use in a range of applications, including research and development, competitions, and the service industry. A reduced platform cost was an essential consideration in our design. We introduce a specialized and inexpensive mechanical design, which includes a parallel-kinematics leg design, external gears, and low-cost controllers and sensors. The humanoid robot is equipped with an efficient electronic structure and a tablet computer for task scheduling, control, and perception, as well as an embedded controller for solving forward & inverse kinematics and low-level actuator control. The perception system recognizes objects at real-time inference with Deep Learning-based detection algorithms without a dedicated GPU. We present and evaluate the capabilities of our newly developed advanced humanoid robot and believe it is a suitable platform for the academic and industrial robotics community.

Recent advances in humanoid robotics have opened up many new directions for humanoid robotics research. Multimedia and arts are applications with significant growth opportunities. In this paper, we introduce magic shows as a benchmark and research problem for humanoid robots and discuss our flexible and versatile system architecture. The goal is to have a humanoid robot perform as a magician in front of an audience. Note that we focus on magic performances that require complex manipulation and interaction of the robot instead of tricks where the robot is passive and used mainly as a prop. So in our tricks, the magic is critically dependent on the skills of the humanoid robot. Furthermore, a successful magic show requires that the robot interacts meaningfully and goal directed with the audience. We describe the technical details of a pen and drawing magic trick, the human-robot collaboration, hardware architecture, and software architecture of our system. One interesting aspect of our system is the inclusion of various personalities for the robot magician. Overall, the robot performed the magic tricks well in the competition and won third place at the IEEE IROS Humanoid Application Challenge(HAC) 2019 in Macau.

Reinforcement Learning (RL) is a powerful tool and has been increasingly used in continuous control tasks such as locomotion and balancing in robotics. In this paper, we tackle a balancing task in a highly dynamic environment, using a humanoid robot agent and a balancing board. This task requires complex continuous actuation in order for the agent to stay in a balanced state. In this work, we propose an RL algorithm structure based on the state-of-the-art Proximal Policy Optimization (PPO) using GPU-based implementation; the agent achieves successful balancing in under 40 min of real-time. We sought to examine the impact of action space shaping on sample efficiency and designed 6 distinct control modes. Our constrained parallel control modes outperform the naive baseline in both sample efficiency and variance to the starting seed. The best-performing control mode, using parallel configuration, including lower body and shoulder roll joints named (PLS-R), is 33% more sample efficient than all the other defined modes, indicating the impact of action space shaping on the sample efficiency of our approach.Our implementation is open-source and freely available at: https://github.com/NTNU-ERC/Robinion-Balance-Board-PPO .

This paper discusses challenges and opportunities when using competitions in robotics education. The authors describe the Federation of International RoboSports Association (FIRA) competition and, in particular, the FIRA Youth—Mission Impossible, an event targeted at overcoming problems of students consciously or thoughtlessly plagiarizing, by forcing students to solve previously unknown tasks. The paper shows an example of the positive influence robot competitions can have on cutting-edge research, even when targeted at younger roboticists. The FIRA Youth—Mission Impossible 2022 competition, where students had to measure the weight of bottles, and hence the wrench applied on the robot using only proprioception, inspired an initial approach and the creation of a practical test-bed for much more complex wrench estimation on hexapod robots.

This paper describes a novel approach to introduce early undergraduate and high schools to humanoid robotics. Instead of building a full humanoid robot, we start with the construction of a single leg, and analyze its kinematic and dynamic behavior. The approach is heavily based on writing various programs to solve and visualize the domains. We introduce and derive fundamental principles of the forward and inverse kinematics of a humanoid robot. The inverse kinematics are approximated using the Jacobian pseudo-inverse method. The control of the motion of a robot is introduced using the versatile PID controller. We designed two new challenge problems to motivate students with our exposition. First, a marble maze game is mounted on the leg and using their understanding of the kinematics, students implement a program to roll the ball from the start to the goal area. Secondly, we implement 1D and 2D ball balancing challenges for a ball on a plane. The paper finishes by introducing a set of modern web technologies that can be used to implement portable and efficient real-time control and simulation of robot systems.

Balancing a two-wheeled scooter is considered a challenging task for robots, as it is a non-linear control problem in a highly dynamic environment. The rapid pace of development of deep reinforcement learning has enabled robots to perform complex control tasks. In this paper, a deep reinforcement learning algorithm is proposed to learn the steering control of the scooter for balancing and patch tracking using an unmodified humanoid robot. Two control strategies are developed, analyzed, and compared: a classical Proportional–Integral–Derivative (PID) controller and a Deep Reinforcement Learning (DRL) controller based on Proximal Policy Optimization (PPO) algorithm. The ability of the robot to balance the scooter using both approaches is extensively evaluated. Challenging control scenarios are tested at low scooter speeds, including 2.5, 5, and 10 km/h. Steering velocities are also varied, including 10, 20, and 40 rad/s. The evaluations include upright balance without disturbances, upright balance under disturbances, tracking sinusoidal path, and path tracking. A 3D model of the humanoid robot and scooter system is developed, which is simulated in a state-of-the-art GPU-based simulation environment as a training and test bed (NVidia's Isaac Gym). Despite the fact that the PID controller successfully balances the robot, better final results are achieved with the proposed DRL. The results indicate a 52% improvement on average in different speeds with better performance in path tracking control. Controller command evaluation on the real robot and scooter indicates the robot's complete capability to realize steering control velocities.

Abstract The field of humanoid robotics is constantly evolving, with new advances creating exciting opportunities for research and development. Especially in the entertainment area, robotics applications show significant growth potential. To guide and track the progress of robotics research, good benchmark problems are crucial, but especially investigating human–robot interaction capabilities is difficult. This paper examines robot magic shows as a benchmark and research direction for entertainer robots and discusses flexible and versatile system architectures for robot magicians in the humanoid robot application challenge competition since 2017. The goal is a detailed analysis of magic tricks and magician robots presented in front of audiences. This paper reviews the hardware components and examines the robot platforms of participating teams in the robot magic show, the types of magic performances, the magic tricks and tools, the algorithms for the magic tricks, and the framework for humanoid robots. By providing a comprehensive analysis of these elements, we can gain insight into the capabilities of advanced humanoid robots for high-performance magic tricks. In conclusion, this paper highlights the exciting potential of humanoid robotics and entertainment fusion. In addition, we analyze the use of humanoid robots in magic shows, presenting the industrial potential of entertainer robots.

The Covid-19 pandemic has affected many lives over the past year. In addition to the enormous health cost, the necessary lockdowns and government-mandated suspension to prevent the spread of the virus had a huge economic impact. The new challenges in 2021 were combating new virus mutations and providing effective vaccines globally. Artificial intelligent (AI) and machine learning have made significant improvements in many different applications during the last decades. One of the advanced and robust technologies in machine learning is deep learning (DL), which can be employed to help prevent initial infections and detect and monitor their progress and side effects. Fast and accurate Covid-19 infection detection and treatment of suspected patients is essential to make better decisions, ensure treatment, and even save patients' lives. Modern technologies are required to achieve these objectives and create a sustainable society. This article presents a taxonomy in DL algorithms to cover both the technical novelties and empirical results techniques for Covid-19 in smart cities. In this regard, (i) we demonstrate possible DL algorithms capable of combating Covid-19; (ii) we propose an up-to-date perspective of DL algorithms in social prevention and medical treatment; and (iii) we identify the challenges in combating Covid-19 outbreaks.

The field of robotics, and more especially humanoid robotics, has several established competitions with research oriented goals in mind. Challenging the robots in a handful of tasks, these competitions provide a way to gauge the state of the art in robotic design, as well as an indicator for how far we are from reaching human performance. The most notable competitions are RoboCup, which has the long-term goal of competing against a real human team in 2050, and the FIRA HuroCup league, in which humanoid robots have to perform tasks based on actual Olympic events. Having robots compete against humans under the same rules is a challenging goal, and, we believe that it is in the sport of archery that humanoid robots have the most potential to achieve it in the near future. In this work, we perform a first step in this direction. We present a humanoid robot that is capable of gripping, drawing and shooting a recurve bow at a target 10 meters away with considerable accuracy. Additionally, we show that it is also capable of shooting distances of over 50 meters.