This study presents a novel method for estimating grasp stability using a soft robotic gripper before grasping an object. Soft robotic grippers have attracted attention for their ability to grasp objects of various shapes due to their high adaptability. However, their flexibility can lead to unstable grasping, which may cause the object to be dropped. Therefore, estimating whether the soft gripper can grasp an object stably beforehand is crucial. In this study, a camera is used as an external sensor to capture both the object shape and gripper position, which are important factors in the gripper’s deformation. The contour information of the object extracted from the images is primarily utilized. First, to describe the gripper deformation from images, the gripper is modeled by discretizing it into a serial chain of rigid bodies connected by spring joints, assuming that the bending angle and deformation follow constant curvature. Next, using the contours of both the gripper and the object, the contact position between them is predicted before grasping. The contact forces are then estimated based on the deformation results, taking into account the rotation of the spring joints in the discretized model. These methods are subsequently integrated, enabling grasp stability estimation before grasping an object. In the experiment, a strong correlation was observed between the grasp stability predicted by the proposed method before grasping and the actual grasp stability measured during grasping.

This paper presents a novel method for grasping objects with varying stiffness using an underactuated hand and a stereo camera. In factories, robots are required to handle a wide variety of objects. Tasks such as grasping soft objects without causing damage are particularly important in industries like food processing. While many existing approaches equip robotic hands with sensors, such as force or pressure sensors, these methods are unsuitable for food items due to hygiene concerns. To address the challenges of grasping various objects without causing damage or dropping them, underactuated hands that can conform to object shapes have gained attention. In this study, we propose a method for controlling an underactuated hand using only a stereo camera as an external sensor. First, the target object is detected using a background subtraction method. Next, the contact between the hand and the object is detected. Then, the object is grasped with appropriate force, calculated based on four elements: the centroid shifts of the hand and the object, the deformation rate of the object, and the occlusion rate of the hand. Finally, drop detection is performed to ensure the object is not dropped during pick-and-place tasks. Experiments were conducted using six different objects to validate the proposed method.

Soft robotic grippers are highly adaptable to various objects because they can deform and fit object shapes. However, grasping stability may change owing to the posture of the gripper while grasping an object. For a stable grasp, it is necessary to estimate the grasping posture before the grasp, namely pre-touch estimation. In particular, for soft robotic grippers, an important factor in the grasping posture is gripper deformation. In previous studies, pre-touch estimation was researched only for rigid grippers without considering deformation, and the stability of grasping an object using soft grippers was evaluated after grasping. The deformation of the gripper depends on the intrinsic characteristics of the gripper deformation (e.g., stiffness) and the contact positions between the gripper and object, that is, how the gripper can deform and where on the gripper is in contact with the object. Deformation characteristics vary from one gripper to another, and the contact positions change according to the characteristics, gripper location, and object shape. Thus, an estimation method that considers these conditions is required to achieve a pre-touch estimation of the deformation of soft robotic grippers.This study presents a vision-based method for estimating the deformation of a soft robotic gripper prior to grasping an object. The entire method is performed before the gripper grasps an object. In the first process, the deformation model that shows the manner in which the gripper can deform is defined using three approaches: discretization of the gripper based on a model of a serial chain of rigid bodies connected with a spring joint, the bending angle of the entire gripper, and piecewise constant curvature. Next, using an image, the bending angle of the entire gripper is acquired to calibrate the deformation model. Subsequently, the contact points between the gripper and object are predicted by obtaining their contours from an image. Finally, the deformation of the entire gripper is estimated based on the deformation model and predicted contact points. Three experiments were conducted to evaluate the accuracy and versatility of the proposed method with respect to gripper location and object shape.

This paper presents a novel method to control an underactuated hand by using only a monocular camera, not using any internal sensors. In food factories, robots are required to handle a wide variety of foods without damaging them. To accomplish this, the use of underactuated hands is effective because they can adapt to various food shapes. However, if internal sensors such as tactile sensors and force sensors are used in the underactuated hands, it may cause a problem with hygiene and require complicated calibration. Moreover, if external sensors such as cameras are used, it is necessary to grasp foods without damaging them by using external information such as images. In our method, to tackle these problems, a camera is used as an external sensor. First, contact between the hand and the object is detected by using the contours of both, obtained from a camera image. Then, to avoid damaging the object, the following information is extracted from camera images and observed: the centroid of both the hand and object, the deformation of the object, and the occlusion rate of the hand. Furthermore, to prevent the object from dropping while the robotic arm is in motion, the distance between the centroid of the hand and the object is calculated. The experiments were conducted using twelve different food items.

In this paper, we propose an image feedback control for an underactuated hand to grasp brittle/deformable food without damaging it. Automation using robots in food factories are being promoted to compensate labor shortages and improve productivity. Since food products vary in shape and hardness and are difficult to be grasped by conventional robots, robot hands that conform to the shape of the object have been developed. In this research, we aimed to control an underactuated hand by using a monocular camera, without sensor or marker. In our method, we detect the deformation of the object using optical flow, adjust appropriate grasping force using area information of robot hand, and detect the grasping state using the relative displacement of the object and the hand. Through the experiments, twelve types of real foods were successfully grasped without dropping and without damaging expect two foods.

In-hand manipulation to translate and rotate an object is a challenging problem for robotic hands. As one solution, robotic hand with belts around fingers ( $active~surfaces$ ) has been developed for continuous and seamless manipulation. However, in practice, the grasped object can only be rotated through a small range less than 90° except the objects with simple shapes like cubes and cylinders. This is because the fingers cannot follow the width required not to drop the object or the desired rotation cannot be produced depending on its shape, leading to dropping the object or unable to rotate it anymore. This paper presents a method to address these problems and rotate objects of various shape and sizes through a large range of motion. A stereo camera is attached to a two-fingered robotic hand with belts. The changes in the contact points between the surfaces of the belts and object are predicted. Based on the prediction, the belts are controlled to adjust the angular velocity of the object such that the fingers can follow the width required to grasp it and the appropriate rotation can be produced. The fingers are controlled to follow the predicted contact points and deflect the belts to both cancel the unwanted rotation and generate the desired rotation. Through experiments in which 32 objects of 16 shapes and 2 sizes, and other real-world objects were rotated to 1 revolution, the rotational ranges for various objects were larger than in the other studies, confirming the validity of the proposed method.

In this paper, we propose the principle of the robot hand equipped with mechanisms consisted of a belt and a spatula at the fingertips to scoop and pull-in a soft object in addition to simply pinching up an object. This fingertip mechanism allows the fingertip to be inserted between a target object and floor or other objects with low friction and enables stable gripping by pulling the scooped object into the robot hand. With these functions, the robot hand can also pick up a soft object without causing the object excessive deformation or breakage. To achieve both scooping and pulling-in functions compactly, the belt is driven by a mechanism using a motor and flat spiral spring. Furthermore, this paper proposes a small linear drive for moving the spatula linearly that uses a leaf spring with holes in a row as a rack gear. At the end of this paper, we discuss the configurations of the actual machine and the knowledge of the pinching up motion and scooping and pulling-in function obtained from basic experiments with the actual robot hand.

This paper presents a vision-based control to pick up an object by a robotic hand. In-hand manipulation, i.e., changing the position and orientation of the grasped object without dropping it, is a challenging task. It is important and necessary for tasks like achieving a stable grasp by translating or rotating the grasped object even though only a part of the object is grasped. However, it is difficult for soft objects like food, because the grasping force needs to be just enough to hold and manipulate the object without crushing it. To tackle this problem, we propose a system which integrates the control of the hand and object detection via a camera. The target robotic hand is configured by two parallel grippers with conveyor belts for manipulation, and is equipped only with a stereo camera as a sensor. From the camera image frames, the three-dimensional position, orientation, and size of the object are calculated. While controlling the grippers and belts, the slippage between belts and the object is estimated based on the difference between their displacements. The validity of the proposed system is verified through the experiments wherein the hand is controlled to pick up objects of varying size and softness. By experimentally evaluating whether the objects have slipped, been crushed, or been manipulated to the target position, it was confirmed that all objects could be picked up with the appropriate force without dropping or crushing them.

In-hand manipulation (IHM) is an important ability for robotic hands. This ability refers to changing the position and orientation of a grasped object without dropping it from the hand workspace. One major challenge of IHM is to achieve a large range of manipulation (especially rotation), regardless of the shape, size, and the orientation during manipulation of the grasped object. There are two main challenges - the manipulation range (due to the range of motion of the hand) and keeping the object grasped under all shapes and orientations. Specifically, even when the contact points between the hand and the object switch and the positions of these points change due to its shape and changing orientation, constant grasp of the object is required. This paper presents an IHM method for a robotic hand with belts, based on the prediction of the contact-point changes via image information. The focus is on a robotic hand that has a two-fingered parallel gripper with conveyor belts which can continuously manipulate an object through a large range. A stereo camera is attached to the hand. First, the contour of the grasped object is acquired from the camera. From the contour, the switching of the contact points between the surfaces of the belts and the object is predicted. Then, the positions of the contact points in the next frame are estimated by rotating the contour. The velocities of the belts are calculated based on the prediction of the switching. The fingers are controlled to follow the estimated positions of the contact points, via a feed-forward control. The effectiveness of the proposed method is verified through in-hand manipulation experiments for 22 objects of various shapes and sizes.

Abstract This paper presents a vision-based in-hand manipulation method. Picking up an object from a pile is an important task for a robotic hand. In particular, for objects of various sizes, weights, and hardness, it is necessary to detect slippage between the object and the surface of the hand. Meanwhile, dexterously changing the position and orientation of the grasped object in the hand workspace is also necessary for sequential pick-and-place tasks. In an effort to achieve sequential motion, an algorithm for picking up and translating a desired object is proposed. A two-jaw parallel gripper with a conveyor belt on each gripper surface is used, which is capable of both translating and rotating a grasped object. In order to observe the slippage, the actuation of the belts is combined with the detection of the manipulation status from images captured by a stereo camera attached to the hand. With an algorithm for the combination, both picking up and translating various objects of unknown sizes and hardness are achieved without needing machine learning models (which require large amounts of training data) or any kind of tactile sensing. The validity of the proposed method is verified through experiments to pick up various objects and translate them to the given position.

This paper presents an occlusion-handling method for a target-tracking robot with a stereo camera. One of the main challenges with the robot is to continue tracking when the illumination changes and occlusion occurs. In order to cope with the challenge, we use both color and disparity images acquired from a stereo camera. The tracking system is composed of three phases: candidate extraction, target identification, and occlusion handling. First, by using only three-dimensional (3D) information, target candidates are extracted. Second, the target is identified from the candidates based on a combination of both color and location features of the target and candidates. The combination depends on illumination changes that are supposed by changes in the white balance. Finally, the state of occlusion is estimated by results of both the analysis of the positional relationship between the candidates and the identification of a target. The proper procedure for the state is implemented. In the off-line experiments, the proposed method is compared with previous methods. Then, the proposed method is applied to a mobile robot, and an on-line experiment is carried out. Through the experiments, the effectiveness of the proposed method is verified.

This paper addresses a target-tracking method for a mobile robot with the purpose of resolving the occlusion problem. The approach is based on both color and disparity images acquired from a stereo camera. To improve the robustness of the method against occlusion, two new techniques are adopted. First, human candidates are detected based on measured 3D points. This allows the system to detect people who are partly occluded. Second, the occlusion-detection technique is used to carry out tracking procedures appropriate to occlusion states. According to the positional relationship between a target and persons/objects, three occlusion states are defined. The effectiveness of the proposed method is tested through targettracking experiments in real-world environments. The experimental result is compared with our previous targettracking method, which is not equipped with any occlusion handling technique. The comparative robustness of the proposed method is demonstrated.

In this paper, we propose a target-tracking system for a mobile robot equipped with a stereo camera. Mobile service robots with the ability to track a specific person in dynamic environments have been required. In such environments, varying illumination and the presence of multiple people are challenges to carrying out target tracking. Color and location information is used for the target's features, that are useful for distinguishing a target from the other people. However, color information is not resilient to illumination changes. On the other hand, location information might be infeasible when non-target people present in the environments. Therefore, it is necessary to combine each information according to the situations, in which either information is infeasible to use as the feature. In order to make the system robust to varying illumination and presence of non-target people, a parameter of illumination changes is introduced in this paper. The parameter is defined using automatically adjusted white balance. The color and location features are weighted based on white-balance changes to determine a target. The effectiveness of the proposed system is verified through target-tracking experiments in outdoor environments. It is demonstrated that the proposed method can successfully recognize a target in the environments where the lighting condition changes extremely and non-target people present.

Target tracking is one of the important functions for autonomous mobile robots. In the environments where a target-tracking system is used, there may be many people and objects. When the system is applied in such environments, occlusion is a challenge for achieving target tracking. In this paper, an occlusion handling method for target tracking with a mobile robot is proposed. By using a stereo camera, both color and location information is used for target detection. During target tracking, the state of occlusion is determined. According to the state, an appropriate tracking process is applied. The effectiveness of the proposed method is verified through target-tracking experiments. In the experiments, the proposed system is compared with the target-tracking system without the occlusion handling method.

Tracking a specific person in dynamic environments is a fundamental task of mobile service robots. Image information is essential to identify a target person, however, the information is not reliable under varying illumination. In this paper, we propose a target-tracking system using a stereo camera. Color and location information is used for the target's feature, which is useful to distinguish a target from the other people. An evaluation value to identify a target is defined as weighted sum of the color and location features. The weight to the features is derived from a parameter of illumination changes. The parameter of illumination changes provides the system with capability of robust tracking even under varying illumination. We confirmed robustness of the proposed system through target-tracking experiments in outdoor environment where the lighting condition changes extremely.

This paper presents a target tracking system for a mobile robot in both indoor and outdoor environments. A stereo camera, which is robust to sunlight and illumination changes, is used. Human regions in images are detected from 3D information. Using color information, a target region is discriminated from the detected human regions. Hue and saturation are chosen as features robust to illumination changes. Finally, a mobile robot is controlled based on the 3D information of the detected target region. The effectiveness of the proposed system is verified through human following experiments in both indoor and outdoor environments.

This paper presents a target tracking system for a mobile robot in both indoor and outdoor environments. A stereo camera, which is robust to sunlight and illumination changes, is used. Human regions in images are detected from 3D information. Using color information, a target region is discriminated from the detected human regions. Hue and saturation are chosen as features robust to illumination changes. Finally, a mobile robot is controlled based on the 3D information of the detected target region. The effectiveness of the proposed system is verified through human following experiments in both indoor and outdoor environments.