Liver registration by overlaying preoperative 3D models onto intraoperative 2D frames can assist surgeons in perceiving the spatial anatomy of the liver clearly for a higher surgical success rate. Existing registration methods rely heavily on anatomical landmark-based workflows, which encounter two major limitations: 1) ambiguous landmark definitions fail to provide efficient markers for registration; 2) insufficient integration of intraoperative liver visual information in shape deformation modeling. To address these challenges, in this paper, we propose a landmark-free preoperative-to-intraoperative registration framework utilizing effective self-supervised learning, termed Self-P2IR. This framework transforms the conventional 3D-2D workflow into a 3D-3D registration pipeline, which is then decoupled into rigid and non-rigid registration subtasks. Self-P2IR first introduces a feature-disentangled transformer to learn robust correspondences for recovering rigid transformations. Further, a structure-regularized deformation network is designed to adjust the preoperative model to align with the intraoperative liver surface. This network captures structural correlations through geometry similarity modeling in a low-rank transformer network. To facilitate the validation of the registration performance, we also construct an in-vivo registration dataset containing liver resection videos of 21 patients, called P2I-LReg, which contains 346 keyframes that provide a global view of the liver together with liver mask annotations and calibrated camera intrinsic parameters. Extensive experiments and user studies on both synthetic and in-vivo datasets demonstrate the superiority and potential clinical applicability of our method. The code and dataset are available at https://github.com/junzastar/Self-P2IR.

Identification of ancient wood is crucial for historical research and the preservation of cultural heritage. This study focuses on identifying three ancient wood samples from the Ming Dynasty of China using terahertz (THz) technology and a convolutional neural network (CNN). A comparative analysis was conducted to identify these unknown ancient woods by comparing them with three types of modern pine and three types of modern fir. The terahertz absorption coefficients of nine types of wood samples were first calculated, followed by an analysis of their frequency characteristics within the 0.01-2.5 THz spectral range. The THz spectra were then preprocessed using wavelet denoising (WD) and low-pass filtering (LPF). Dimensionality reduction was subsequently applied to the spectral data based on a cumulative variance contribution threshold of 95%. Finally, the CNN model was developed to identify ancient wood species by minimizing root mean square error (RMSE). Results demonstrate that ancient wood samples THKM7 and NTGSMO1 share five characteristic frequencies with modern pine and fir and a consistent upward trend in absorption coefficients. In addition, the absorption coefficient of ancient wood NTSO3 shows significant deviations in frequency points and amplitudes. Furthermore, the CNN prediction results reveal the minimal RMSE values between the ancient wood THKM7 sample and modern pine XS sample (RMSE = 0.0143) and between the ancient wood NTGSMO1 sample and modern fir LS sample (RMSE = 0.0265). Finally, the accuracy of the prediction results was verified by Generalized Regression Neural Network (GRNN) and Random Forest (RF) classifiers. The study integrating THz technology with deep learning provides a research idea for ancient wood identification and can advance scientific research in cultural heritage conservation.

Securely harvesting apples without damage remains a challenge owing to their softness, vulnerability, and irregular shapes. In this study, a flexible swallowing (FS) gripper was designed and tested for harvesting apples. To analyze the adaptive grasping process of this gripper, first, a closed equation of the force-to-deformation model for the finger was proposed based on the Chained-Beam-Constraint Model (CBCM). The correctness of the model was verified by finite element analysis (FEA), and the maximum error was no more than 1.7 %. Subsequently, the grasping force sensing model related to bending angle of the finger was established based on the force-to-deformation model. Finally, taking an apple with a diameter of 80 mm as the experimental object, several groups of grasping tests were conducted with the gripper. The bending angle and force sensing error for different grasping positions of the fingers were analyzed and compared. The results demonstrated that the gripper can sufficiently perceive the grasping force, and the force sensing accuracy in the middle of the finger was the best, with an average absolute error and relative error of 0.153 N and 5.65 %, respectively; A better envelope ability and greater grasping force was obtained when grasping with the bottom of the finger; the maximum bending angle and maximum contact force were 22.6 degrees and 5.72 N respectively. Moreover, a harvesting experiment using this gripper installed at the end of a robot arm was conducted, which further verified that this gripper has a good grasping ability for apples and can be sufficiently applied in actual robot harvesting.

Existing target detection models are large and have multiple network parameters, which can severely slow down the detection speed when deployed on small, low-cost GPU-free Industrial Personal Computers (IPC). As a result, this study proposes a lightweight real-time tomato detection and point-picking integrated network model based on YOLOv5 (TDPPL-Net). Firstly, the algorithm replaces the YOLOv5 backbone with a four-group lightweight downsampling model consisting of Ghost Conv and Ghost Bottleneck to reduce the model size, while adding the attention mechanism SimAM module to improve detection accuracy after each scale’s feature map. Secondly, the Spatial Pyramid Pooling-Fast (SPPF) network structure is used and the convolutional layers in the (Feature Pyramid Network and Path Aggregation Network) FPN+PAN structure are replaced with a depth-separable convolution to reduce the computational effort. Finally, the center of the bounding box is used as the picking point, and the corresponding depth information is obtained in combination with the Intel RealSense D435 camera, which is converted into 3D coordinates under the robot arm coordinate system after hand-eye calibration. The experimental results show that TDPPL-Net reduces the number of parameters by 59.84% compared with the original YOLOv5, the model volume is only 40% of the original, the mAP is 93.36, and the real-time detection speed on the IPC without GPU acceleration is 31.41 FPS, which is 170.31% higher than YOLOv5. The TDPPL-Net increases detection speed on low-performance equipment without compromising detection accuracy. It can detect and locate tomato picking points in real-time in the complex natural environment, which can meet the working requirements of harvesting robots.

In the field of computer vision, image saliency target detection can not only improve the accuracy of image detection but also accelerate the speed of image detection. In order to solve the existing problems of the saliency target detection algorithms at present, such as inconspicuous texture details and incomplete edge contour display, this paper proposes a saliency target detection algorithm integrating multiple information. The algorithm consists of three processes: preprocessing process, multi-information extraction process, and fusion optimization process. The frequency domain features of the image are calculated, the algorithm calculates the frequency domain features of the image, introduces power law transform and feature normalization, improves the frequency domain features of the image, saves the information of the target region, and inhibits the information of the background region. On three public MSRA, SED2, and ECSSD image datasets, the proposed algorithm is compared with other classical algorithms in subjective and objective comparison experiments. Experimental results show that the proposed algorithm can not only accurately and comprehensively extract significant target regions but also retain more texture information and complete edge information while satisfying the human visual experience. All evaluation indexes are significantly better than the comparison algorithm, showing good reliability and adaptability.

To meet the demand for canopy morphological parameter measurements in orchards, a mobile scanning system is designed based on the 3D Simultaneous Localization and Mapping (SLAM) algorithm. The system uses a lightweight LiDAR-Inertial Measurement Unit (LiDAR-IMU) state estimator and a rotation-constrained optimization algorithm to reconstruct a point cloud map of the orchard. Then, Statistical Outlier Removal (SOR) filtering and European clustering algorithms are used to segment the orchard point cloud from which the ground information has been separated, and the k-nearest neighbour (KNN) search algorithm is used to restore the filtered point cloud. Finally, the height of the fruit trees and the volume of the canopy are obtained by the point cloud statistical method and the 3D alpha-shape algorithm. To verify the algorithm, tracked robots equipped with LIDAR and an IMU are used in a standardized orchard. Experiments show that the system in this paper can reconstruct the orchard point cloud environment with high accuracy and can obtain the point cloud information of all fruit trees in the orchard environment. The accuracy of point cloud-based segmentation of fruit trees in the orchard is 95.4%. The R2 and Root Mean Square Error (RMSE) values of crown height are 0.93682 and 0.04337, respectively, and the corresponding values of canopy volume are 0.8406 and 1.5738, respectively. In summary, this system achieves a good evaluation result of orchard crown information and has important application value in the intelligent measurement of fruit trees.

Hardness is an important physical property of fruits and vegetables that directly affects the effectiveness of the picking manipulators of robots. Therefore, a method for recognizing the hardness of fruits and vegetables based on tactile array information is proposed. This allows the picking manipulator to recognize the hardness properties of fruits and vegetables, thus aiding the robot to stably grasp these products without damage during picking. First, an experimental platform for the tactile information acquisition system of the manipulator was built to collect the original dataset of the tactile sequence generated as the manipulator grasps the fruits and vegetables in real time. Then, two classification models for recognizing the hardness of fruits and vegetables were formulated and tested. In these two proposed models, the sample feature set obtained after the dimensionality reduction processing by the principal component analysis (PCA) was used to train and test the two classifiers based on k-nearest neighbor (KNN) and support vector machine (SVM) algorithm. The classification accuracy rates of the PCA-KNN and PCA-SVM are 90.03% and 94.27%, respectively, indicating that the accuracy of the latter is significantly better than that of the former. Finally, an online grabbing recognition experiment using the manipulator was implemented to verify the practicability of the PCA-SVM classifier. The accuracy rate of online recognition reached 90%, which is a noteworthy experimental result.

To meet the demand for intelligent measurements of canopy morphological parameters, a mobile LiDAR scanning system with LiDAR and IMU as the main sensors was constructed. The system uses a LiDAR-IMU tight coupling odometry method to reconstruct a point cloud map of the area surveyed. After using the RANSAC algorithm to remove the map ground, the European clustering algorithm is used for point cloud segmentation. Finally, morphological parameters of the canopy, such as crown height, crown diameter, and crown volume, are extracted using statistical and voxel methods. To verify the algorithm, a total of 43 trees in multiple plots of the campus were tested and compared. The algorithm defined in this study was evaluated with manual measurements as reference, and the morphological parameters of the canopy obtained using the LOAM and LeGO-LOAM algorithms as the basic framework were compared. Experiments show that this method can be used to easily obtain the crown height, crown diameter, and crown volume of the area; the correlation coefficients of these parameters were 0.91, 0.87, and 0.83, respectively. Compared with the LOAM and LeGO-LOAM methods, they were increased by 0.004, 0.12, and 0.13 and 0.07, 0.15, and 0.04, respectively. The test results for this new system are positive and meet the requirements of horticulture and orchard measurements, indicating that it will have significant value as an application.

Accurately estimating the peak cutting force for cutting citrus fruit stems is helpful for the design of the end-effector of a harvesting robot and improves the harvesting success rate of the harvesting robot. In this study, five factors that influence the cutting of citrus fruit stems with a harvesting robot in a natural environment were analyzed, including cutting speed (v), the gap between the two cutting blades of the end-effector (s), the diameter of the stem (d), and the deflection (theta(def)) and inclination (theta(inc)) angles of the stem. An experimental platform was built for cutting citrus fruit stems with the blade gap (s) as a fixed value. A mechanical model for calculating the maximum cutting force was established based on the partial least squares method, and d, v, theta(def), and theta(inc) were the independent variables of the model and were verified by repeated tests. According to the test results, the mechanical model was modified when the citrus fruit stem diameter was greater than or equal to 3.2 mm, and the modified model was also verified by tests. The verification results indicated that, compared with the actual peak force for cutting citrus fruit stems, the percentage error of the peak force calculated with the model was 11.51% when the stem diameter was less than 3.2 mm and 4.94% when the stem diameter was greater than or equal to 3.2 mm, which can provide a significant reference for evaluating the peak force for cutting citrus fruit stems and designing the end-effector of citrus harvesting robots.

We present a real-time Truncated Signed Distance Field (TSDF)-based three-dimensional (3D) semantic reconstruction for LiDAR point cloud, which achieves incremental surface reconstruction and highly accurate semantic segmentation. The high-precise 3D semantic reconstruction in real time on LiDAR data is important but challenging. Lighting Detection and Ranging (LiDAR) data with high accuracy is massive for 3D reconstruction. We so propose a line-of-sight algorithm to update implicit surface incrementally. Meanwhile, in order to use more semantic information effectively, an online attention-based spatial and temporal feature fusion method is proposed, which is well integrated into the reconstruction system. We implement parallel computation in the reconstruction and semantic fusion process, which achieves real-time performance. We demonstrate our approach on the CARLA dataset, Apollo dataset, and our dataset. When compared with the state-of-art mapping methods, our method has a great advantage in terms of both quality and speed, which meets the needs of robotic mapping and navigation.

Moving and operating autonomously in a field or orchard environment is challenging for agricultural robots due to the complex task requirements and highly unstructured conditions. The human-computer interaction-based remote control can provide the robots with an alternative solution to assist decision making and motion planning. In this study, a virtual reality (VR) and Kinect-based immersive teleoperation system were proposed to connect the physical and virtual world by utilizing real-time large-scale unstructured agricultural environment reconstruction and simultaneous virtual environment creation. The proposed system, with a relatively large server, can convert the scene into a realistic model by combing the depth and color image streams received from Kinect, and project them back into 3D (three dimensions) space in such a manner that the real 3D scene inside the camera's field of view is recreated virtually. To create a VR environment for a VR headset, an optimized Bundlefusion-based algorithm was developed for real-time 3D reconstruction of the unstructured agricultural scene in the natural environment. Additionally, the performance of the proposed real-time 3D reconstruction algorithm was evaluated and compared with Bundlefusion and voxel hashing algorithms in different large-scale unstructured agricultural environments. Performing optimizing pose on our optimized algorithm leverages a large number of processing cores available to minimize the delay between data capture and rendering, and it reduces the average acquired time to process each frame no more than 0.9 ms. The experimental results including, less computer storage occupied, fast frame processing time, and high-quality 3D realistic model indicate that our proposed system and algorithm have the potential applicability of immersive teleoperation in an unstructured agricultural environment.

Grasping, carrying and placing of objects are the fundamental capabilities and common operations for robots and robotic manipulators. Grippers are the most essential components of robots and play an important role in many manipulation tasks, since they serve as the end-of-arm tools, as well as the mechanical interface between robots and environments/grasped objects. Gripper developments are motivated by the great number of different requirements, diverse workpieces and the desire for well adapted and reliable systems. Grippers provide temporary contact with the grasped objects in manipulations. Secure grasping not only requires contacting the objects, but also avoiding the risk of potential slip and damage while the objects are picked and placed. To offer secure grasping for objects with a wide variety of shapes, sizes and materials, various sensors and control strategies are also needed. With the developments of technologies, labor shortage caused by the population aging, as well as the requirements of high automation degree, agricultural robots will find their increasing applications in agricultural and food industries. As the end-of-arm tools for the robots, grippers can be seen as the hands of robots, almost all automatic manipulations are conducted directly by robotic grippers. This paper gives a detailed summary about the state-of-the-art robotic grippers, grasping and sensor-based control methods, as well as their applications in robotic agricultural tasks and food industries. Different from workpiece in industrial environment, agricultural products are fragile and damageable. The requirement for grasping agricultural products is higher than that of grasping of industrial workpieces, various sensors are needed to be installed to the grippers to make them less aggressive, and more flexible and controllable. Therefore, particular attention has been paid to the sensors that used in the grippers to improve their sensing and grasping capabilities. The advantages and disadvantages of the grippers are discussed and summarized. Finally, the challenges and potential future trends of grippers in agricultural robots are reported.

A discrete element model of citrus fruit stalk with a diameter of 3 mm was established using the discrete element method (DEM). The intrinsic parameters and contact parameters of citrus fruit stalk were obtained through measurement. The bonding parameters of the discrete element model of citrus fruit stalk were calibrated through a three-point bending test and shear tests. Based on these parameters, the optimal bonding parameter combination was determined. This included. normal stiffness per unit area 4.6362 x 10(11)N m(-3), shear resistance 1.2549 x 10(11)N m(-3), bonded disc radius 0.222 mm, critical normal stress 9.5742 x 10(7)Pa and maximum shear stress 7.4007 x 10(7)Pa. These optimal parameters were applied to simulate the bending and shearing of a citrus fruit stalk using the established citrus fruit stalk model. Compared to experimental results, the deviation of bending strength and shear peak force were 1.25% and 1.78%, respectively. The results show that a DEM of citrus fruit stalk established can be used to properly simulate the bending and shearing of citrus fruit stalks and that the method of calibrating the parameters for accuracy is reasonable. The results are significant for the study of the mechanical properties of citrus fruit stalks and the optimisation of end-effectors of harvesting robots. (C) 2020 IAgrE. Published by Elsevier Ltd. All rights reserved.

Mechanical harvesting requires agricultural robot to detect fruits automatically. However, effective and accurate detection of cucumber by computer vision system is still a challenge due to similarity between cucumber color and that of branches and leaves, shape irregularity and complex growing environment. To improve the practicability and accuracy of the automatic recognition models, this paper proposed a novel cucumber region detection method using multi-path convolutional neural network (MPCNN), combined with color component selection and support vector machine (SVM). In this method, the cucumber image was transformed into color space to obtain 15 color components and the weight information of relevant features was analyzed by I-RELIEF. In parallel, to remove part of the background area, the OSTU algorithm was applied to segment the G component and Maximally Stable Extremal Regions (MSER) was used to obtain the mask image. In order to maximize the differences between cucumber and leaf, promoting the classification accuracy of SVM, the top three components of the weight were input into the deep learning module to extract and fuse features. In final, cucumber recognition was realized by combining SVM classification with mask image. The recognition results show that more than 90% pixels of cucumber images are correctly classified, and the misidentified pixels are less than 22%. The ratio between the two indicators is over 4, demonstrating the satisfactory performance of the proposed method and highlighting its promising applications in mechanical cucumber harvesting.

This paper presents a method to extract posture from a series sequence of contour images which were taken from the side of a person. We extract a cycle from the image sequence and confirm the body region, then extract the articulation points of a human body in each images of the sequence and normalize the frame number using cubic spline. In the experiment, we used 360 image sequences taken from 60 persons, and the recognition accuracy rate was 90% for groups with 10 people and 80% for groups with 15 people.