Among human perception systems, the haptic system can provide bidirectional energy and information interaction between humans and the environment, which offers unique advantages over visual or auditory systems. Consequently, integrating haptic feedback into teleoperation systems significantly enhances the operator's environmental awareness and task efficiency. Based on two Geomagic Touch haptic devices, a Microsoft Kinect DK sensor, a robot arm, a robot dexterous hand, and a dexterous hand haptic perception module, this paper presents a multimodal teleoperation system that integrates vision and haptic sensation. And then we propose a mapping method between operator hand motion captured at the master-side and teleoperated actions executed at the slave-side, which realizes two-finger-based haptic feedback for the operator. To verify the effectiveness of our proposed system, the hardness discrimination experiment of objects with similar appearance and the recognition experiments of different objects with large differences in appearance have been carried out. The average recognition success rates of 84.3% and 83.3% for these two experiments effectively prove the proposed methods.

Intelligent robots hold broad application prospects in complex, dynamic, and uncertain environments, making precise and real-time environmental perception and interaction capabilities increasingly crucial. This paper presents a vision-based dexterous hand grasping system that integrates object recognition and path planning for robotic manipulation. For target recognition, we propose an HSV-Weighted Fusion Template Matching (HSV-WFTM) algorithm that decomposes HSV channels and performs weighted feature fusion to address the illumination sensitivity of traditional grayscale matching. For motion control, spatial Cartesian interpolation generates stable robotic arm trajectories. Comparative experiments under daylight and artificial lighting conditions demonstrated a 94% overall daytime success rate (96% for sponge tennis ball, 94% for wooden cube, 92% for can), decreasing to 90% at night (92% for sponge tennis ball and wooden cube, 86% for can), confirming the algorithm's adaptability to lighting variations. By enhancing environmental perception and autonomous decision-making, this study provides practical solutions for robotic operations.

Because of the absence of visual perception, visually impaired individuals encounter various difficulties in their daily lives. This paper proposes a visual aid system designed specifically for visually impaired individuals, aiming to assist and guide them in grasping target objects within a tabletop environment. The system employs a visual perception module that incorporates a semantic visual SLAM algorithm, achieved through the fusion of ORB-SLAM2 and YOLO V5s, enabling the construction of a semantic map of the environment. In the human–machine cooperation module, a depth camera is integrated into a wearable device worn on the hand, while a vibration array feedback device conveys directional information of the target to visually impaired individuals for tactile interaction. To enhance the system’s versatility, a Dobot Magician manipulator is also employed to aid visually impaired individuals in grasping tasks. The performance of the semantic visual SLAM algorithm in terms of localization and semantic mapping was thoroughly tested. Additionally, several experiments were conducted to simulate visually impaired individuals’ interactions in grasping target objects, effectively verifying the feasibility and effectiveness of the proposed system. Overall, this system demonstrates its capability to assist and guide visually impaired individuals in perceiving and acquiring target objects.

This paper introduces a novel capacitive sensor array designed for tactile perception applications. Utilizing an all-in-one inkjet deposition printing process, the sensor array exhibited exceptional flexibility and accuracy. With a resolution of up to 32.7 dpi, the sensor array was capable of capturing the fine details of touch inputs, making it suitable for applications requiring high spatial resolution. The design incorporates two multiplexers to achieve a scanning rate of 100 Hz, ensuring the rapid and responsive data acquisition that is essential for real-time feedback in interactive applications, such as gesture recognition and haptic interfaces. To evaluate the performance of the capacitive sensor array, an experiment that involved handwritten number recognition was conducted. The results demonstrated that the sensor accurately captured fingertip inputs with a high precision. When combined with an Auxiliary Classifier Generative Adversarial Network (ACGAN) algorithm, the sensor system achieved a recognition accuracy of 98% for various handwritten numbers from “0” to “9”. These results show the potential of the capacitive sensor array for advanced human–computer interaction applications.

Gesture recognition is still actively researched in non-contact human-computer interaction (HCI), where the Dynamic Time Warping (DTW) algorithm is commonly employed. However, the computation load of traditional DTW algorithms during the matching phase presents a challenge for gesture recognition, especially when the number of reference gestures in the template library increases. In order to solve this problem, a Constraints-based Dynamic Time Warping (CDTW) method is proposed in this paper, including Global-Path Constraint, First-Frame Constraint, and Feature-Vector Constraint. These three constraints are expected to limit the region of the warping path, exclude several reference gestures from the template library, and reduce computation load directly with reduced elements of the feature vector. To verify the proposed CDTW method, Microsoft Kinect-based comparative experiments with the traditional DTW method have been carried out. Experimental results show that our CDTW method boosts the efficiency of gesture recognition with a 17% decrease in recognition time and a 3% increase in average recognition accuracy, compared to the traditional DTW algorithm.

Tactile sensing is a particularly important and challenging task for a modern robot to safely manipulate objects, interact with humans in a shared space, and provide various services. This article presents a 3-D tactile glove for robots with the combination of a piezoresistive-based force sensor array (412 sensors) covering the full hand and a resistive bend sensor array (five sensors) on the back of five fingers. Deep learning-based convolutional neural network (CNN) and multilayer perceptron network (MLP) based methods using the designed tactile glove are proposed for object recognition. In the experiment for recognizing 15 objects with a dexterous robot hand, an average classification accuracy of 93.67% has been achieved. Comparison experiments with three other typical classifiers (the Quadratic support vector machine, weighted KNN, and Bagged Trees) and our MLP and CNN methods show an average recognition accuracy of 91.67% with the 3-D tactile glove, revealing an accuracy improvement of 4.17% over only using the force sensor array. We further apply our 3-D tactile glove and the multimodal CNN to identify three other objects and demonstrate their generalization ability of tactile object recognition with an average success accuracy of 78.33%. The proposed 3-D tactile glove can be further used in human–robot interactions, the design of prosthetics and humanoid robots, and for improving the intelligence level in brain–computer collaborative systems.

Compared with visual and auditory interactions, tactile can provide bidirectional information interaction be-tween the human operator and the environment, which can effectively enhance the quality of human-computer interaction. This paper designed a multi-modal image display system com-bining visual and tactile, where the tactile between the finger and the touchscreen is rendered by adjusting the electrostatic friction. A time-division multiplexing method has been pro-posed for realizing the electrovibration and the finger position tracking with a touchscreen so that it requires no external tracking device. The friction perception threshold experiment and a basic image recognition experiment have been carried out. An average success rate of 93.6% of perception verifies the effectiveness of the proposed system.

Haptic rendering enables people to touch, perceive, and manipulate virtual objects in a virtual environment. Using six cascaded identical hollow disk electromagnets and a small permanent magnet attached to an operator's finger, this paper proposes and develops an untethered haptic interface through magnetic field control. The concentric hole inside the six cascaded electromagnets provides the workspace, where the 3D position of the permanent magnet is tracked with a Microsoft Kinect sensor. The driving currents of six cascaded electromagnets are calculated in real-time for generating the desired magnetic force. Offline data from an FEA (finite element analysis) based simulation, determines the relationship between the magnetic force, the driving currents, and the position of the permanent magnet. A set of experiments including the virtual object recognition experiment, the virtual surface identification experiment, and the user perception evaluation experiment were conducted to demonstrate the proposed system, where Microsoft HoloLens holographic glasses are used for visual rendering. The proposed magnetic haptic display leads to an untethered and non-contact interface for natural haptic rendering applications, which overcomes the constraints of mechanical linkages in tool-based traditional haptic devices.

In the field of multimodal human–computer interaction (HCI), hand (and finger) motion tracking remains a critical challenge because it is the input means of visual reproduction and provides the necessary parameters for real-time modeling of the haptic model. Among motion tracking methods including optical tracking methods, inertial tracking methods, and magnetic motion tracking (MMT) methods, the MMT method has a great development prospect as a marker-based method because of its advantages of no line-of-sight problem, being natural-oriented, and high accuracy. In this article, we first discuss the performance and application scenarios under different configurations of electromagnet-based MMT (EMMT) and permanent-magnet-based MMT (PMMT) systems. And then, the MMT algorithms are reviewed, which can be divided into two broad categories: The model-based algorithms include linear algorithm, nonlinear optimization algorithm, and recursive Bayesian algorithm, and the offline-data-based algorithms include neural network algorithm and lookup table (LUT) algorithm. In addition, a state-of-the-art comparison of MMT algorithms is given. In the end, we have an insightful discussion and give the expected future outlook of MMT.

力触觉再现提供了操作者与虚拟物体间的双向信息和能量交互,有效提高了虚拟现实等应用系统的真实感、沉浸感和操作效率,成为人机交互的新兴技术和研究热点.本文实现了一种新型的基于电磁力控制的二维力触觉再现系统,该系统由二维背景电磁场产生和控制模块、指尖穿戴式永磁铁、人手位置检测模块和中央控制模块组成.基于ANSYS有限元分析,确定了系统中背景电磁铁线圈和指尖永磁铁的最优参数,获得指尖永磁铁所受电磁力与二维可控背景电磁铁驱动电流、指尖电磁铁位置的映射关系,形成离线仿真数据.提出了基于离线仿真数据实时插值的二维力触觉再现中电磁力控制方法.在实现系统原型的基础上,开展了作用力阈值感知基础实验和三维虚拟物体识别实验.实验结果表明,两种三维虚拟物体的识别实验成功率分别为85.7％和71.4％,有效验证了本文所设计力触觉再现方法的有效性.

在人机交互领域中,人手的位置信息往往直接用于交互指令的解读与交互结果的计算,因此高精度的实时人手位置检测是实现非接触式的、自然的人机交互的重要基础.针对Kinect 2.0追踪人体骨骼点获取的三维坐标数据的波动和误差较大的问题,本文提出了基于相关点均值处理的人手位置检测算法.该算法基于深度信息,以手腕为分割阈值点,进行人手图像分割,并对人手位置信息相关点进行空间平均处理与时间平均处理,提高位置检测精度.实验结果表明:基于相关点均值处理的人手位置检测算法是有效的,检测误差在5 mm以内,能够满足在人机交互等应用系统中的基本要求.

近年来,多点交互和多模态融合的力触觉再现系统由于能够进一步提高人机交互的真实感,已经成为该领域的研究热点.基于Leap Motion的多手指位置检测模块、振动触觉再现模块、NRF51822蓝牙通信模块、视觉再现模块和CHAI3D构建的虚拟环境等,设计实现了多点自然交互的、触觉和视觉相融合多模态触觉再现系统.为了验证系统的可行性和有效性,开展了虚拟物体轮廓感知实验和立方体放置任务实验.实验结果表明:虚拟物体轮廓感知实验平均识别率为87.9％;相比单一视觉再现模式,视觉和触觉再现融合模式完成相同任务的时间能节省47.7％.所设计的多点交互触觉再现系统,具有成本低、体积小和控制方法简单等优点,为促进多模态人机交互技术的发展和广泛应用奠定了重要基础.

A recursively indirect quaternion estimator based on the Gaussian particle filter (GPF) is proposed for nonlinear attitude estimation. The key idea is to estimate an on-tangent-plane Gaussian distribution in the GPF scheme for interpreting the uncertainty of the unit quaternion manifold. The unit quaternion is provided with a global nonsingular attitude description in the prediction step, and the three-dimensional attitude error is estimated in the update step. Based on the framework of the GPF, the proposed filter does not need resampling and regularization compared with the PF. The performance of the proposed filter is verified theoretically and evaluated by experiments. The results show that the proposed filter has a faster convergence speed, lower complexity, and lower computational cost than the existing quaternion PF under a comparable accuracy.

Capturing finger joint angle information has important applications in human–computer interaction and hand function evaluation. In this paper, a novel wearable data glove is proposed for capturing finger joint angles. A sensing unit based on a grating strip and an optical detector is specially designed for finger joint angle measurement. To measure the angles of finger joints, 14 sensing units are arranged on the back of the glove. There is a sensing unit on the back of each of the middle phalange, proximal phalange, and metacarpal of each finger, except for the thumb. For the thumb, two sensing units are distributed on the back of the proximal phalange and metacarpal, respectively. Sensing unit response tests and calibration experiments are conducted to evaluate the feasibility of using the designed sensing unit for finger joint measurement. Experimental results of calibration show that the comprehensive precision of measuring the joint angle of a wooden finger model is 1.67%. Grasping tests and static digital gesture recognition experiments are conducted to evaluate the performance of the designed glove. We achieve a recognition accuracy of 99% by using the designed glove and a generalized regression neural network (GRNN). These preliminary experimental results indicate that the designed data glove is effective in capturing finger joint angles.

Tactile sensation is a promising information display channel for human beings that involves supplementing or replacing degraded visual or auditory channels. In this paper, a wrist-wearable tactile rendering system based on electro-tactile stimulation is designed for information expression, where a square array with 8 × 8 spherical electrodes is used as the touch panel. To verify and improve this touch-based information display method, the optimal mode for stimulus signals was firstly investigated through comparison experiments, which show that sequential stimuli with consecutive-electrode-in-active mode have a better performance than those with single-electrode-in-active mode. Then, simple Chinese and English characters and 26 English characters’ recognition experiments were carried out and the proposed method was verified with an average recognition rate of 95% and 82%, respectively. This wrist-wearable tactile display system would be a new and promising medium for communication and could be of great value for visually impaired people.

随着计算机技术发展和移动计算能力的提高,虚拟现实技术得到广泛应用.针对《计算机软件技术基础》理论课程和《计算机实践》实验课程要求,基于微软HoloLens全息智能眼镜和磁力驱动的力触觉交互模块,设计实现了视觉和力触觉融合的多模态虚拟现实实验系统,构建了符合课程教学要求的综合实验平台.基于这一综合实验平台,将计算机最新前沿技术引入课程教学,设计开发了综合性、创新性和开放性的实验项目,以此激发学生学习的积极性和主观能动性,巩固课程知识点,有效促进了学生计算机软件实践能力和创新能力的提高.

实验教学是提高大学生实践能力和创新精神的有效途径.该文以“电子技术”实验课程群为例,利用先进的信息技术和数据处理资源,构建一种过程反馈式实验课程教学新模式,通过实验教学的全过程、实时反馈,有利于持续提高实验教学质量和教学水平.

Current image compression techniques such as JPEG and WebP are widely applied in the age of information. However, lossy compression like JPEG in its nature will introduce visually annoying artifacts while saving Internet bandwidth and storage space. The artifacts such as blocking and ringing are especially sharp at low bitrates. In this paper, we propose a novel dual-residual network to reduce compression artifacts caused by lossy compression codecs. This network directly learns an end-to-end mapping between the distorted image processed by JPEG or other compression methods and the original image, which takes decompressed images with blocking artifacts as input and produces clearer images with less artifacts. Experiments results on the test dataset demonstrate the efficiency of the proposed model, especially at very low bitrates.

Surface electromyography (sEMG) signals are widely used in the recognition of hand gestures. Nowadays, researchers usually increase the number of sEMG signal measurement positions and extract multiple features to improve the recognition accuracy. In this paper, we propose a sEMG measurement position and feature optimization strategy for gesture recognition based on Analysis of Variance (ANOVA) and neural networks. Firstly, four channels of raw sEMG signals are acquired, and four time-domain features are extracted. Then different neural networks are trained and tested by using different data sets which are obtained based on the combination of different measurement positions and features. Finally, ANOVA and Tukey HSD testing are conducted based on the gesture recognition results of different neural networks. We obtain the optimal measurement position sets for gesture recognition when different feature sets are used, and similarly, the optimal feature sets when different measurement position sets are used. Our experimental results show that the feature set of zero crossing and integrated sEMG provides the highest gesture recognition accuracy, which is 94.83%, when four channels of sEMG signals are used; the optimal measurement position set when four sEMG signal features are used for hand gesture recognition is P1+P3+P4, which provides an accuracy of 94.6%.