As the combination of panoramic images and laser point clouds becomes more and more widely used as a technique, the accurate determination of external parameters has become essential. However, due to the relative position change of the sensor and the time synchronization error, the automatic and accurate matching of the panoramic image and the point cloud is very challenging. In order to solve this problem, this paper proposes an automatic and accurate registration method for panoramic images and point clouds of urban large scenes based on line features. Firstly, the multi-modal point cloud line feature extraction algorithm is used to extract the edge of the point cloud. Based on the point cloud intensity orthoimage (an orthogonal image based on the point cloud’s intensity values), the edge of the road markings is extracted, and the geometric feature edge is extracted by the 3D voxel method. Using the established virtual projection correspondence for the panoramic image, the panoramic image is projected onto the virtual plane for edge extraction. Secondly, the accurate matching relationship is constructed by using the feature constraint of the direction vector, and the edge features from both sensors are refined and aligned to realize the accurate calculation of the registration parameters. The experimental results show that the proposed method shows excellent registration results in challenging urban scenes. The average registration error is better than 3 pixels, and the root mean square error (RMSE) is less than 1.4 pixels. Compared with the mainstream methods, it has advantages and can promote the further research and application of panoramic images and laser point clouds.

The accurate extraction of traffic signs is of great significance to the digitization of traffic information and the fine management of traffic. This article introduces an innovative approach to address the challenges associated with recognizing and detecting traffic signs, considering their vulnerability to complex backgrounds, variations in illumination, and motion blur. The proposed method utilizes a semi-supervised learning (SSL) strategy, combining convolutional neural networks (CNNs) with a transformer encoder-decoder architecture, to extract traffic sign features from vehicle panoramic images. To enhance feature extraction, a hierarchical sampling method (HSM) is introduced, which facilitates the extraction of multiscale self-attention features in the transformer encoder-decoder structure. Additionally, a network module called local and global information aggregator (LGIA) is designed based on HSM, enabling the incorporation of both local and global context information. Furthermore, a SSL strategy is adopted to simultaneously train our model using both labeled and unlabeled data samples. This strategy aims to improve the extraction of traffic signs by capitalizing on the broader data set available through unlabeled data. Experimental results demonstrate the effectiveness and robustness of the proposed method in improving the detection and recognition of traffic signs. The approach showcases significant improvements in overcoming the challenges posed by complex backgrounds, variations in illumination, and motion blur. Our approach achieved a 0.9% improvement in the F1-score evaluation over the current classical object detection algorithm on the public data set Tsinghua-Tencent 100K and a 1.1% improvement on the SSW data set.

Accurate and automatic detection of road surface element (such as road marking or manhole cover) information is the basis and key to many applications. To efficiently obtain the information of road surface element, we propose a content-adaptive hierarchical deep learning model to detect arbitrary-oriented road surface elements from mobile laser scanning (MLS) point clouds. In the model, we design a densely connected feature integration module (DCFM) to connect and reorganize feature maps of each stage in the backbone network. Besides, we propose a hierarchical prediction module (HPM) to innovatively use the reorganized feature maps to recognize different types of road surface elements, and thus, semantic information of road surface element can be adaptively expressed on multilevel feature maps. We also add a cascade structure (CS) in the head of model to detect the target efficiently, which can learn the offset between the predicted minimum bounding box of road surface element and ground truth. In experiments, we prove that the proposed method mainly contributed by HPM can maintain robust detection performance, even in the cases of unbalanced category number or overlapping of road surface elements. The experiments also prove that the proposed DCFM can improve the recognition effects of small targets. The CS for predicting boundary offset can detect each target more accurately. We also integrate the designed modules into some rotation detectors, e.g., the EAST and R3Det, and achieve the state-of-the-art results in three road scenes with different categories and uneven distribution of road surface elements, which further shows the effectiveness of the proposed method.

Point clouds of large-scale urban street scenes contain large quantities of object categories and rich semantic information. The semantic segmentation is the basis and key to subsequent essential applications, such as digital twin engineering and city information model. The global feature of point clouds in large-scale scenes can provide long-range context information, which is critical to high-quality semantic segmentation. However, the learning of global spatial saliency considering class label constraints is often ignored in the feature representation of some deep learning models. With regard to this, we propose a Global Feature Self-Attention Encoding (GFSAE) module and a Weighted Semantic Mapping (WSM) module to make the semantic segmentation model of point clouds in large-scale urban street scene focus more on the global salient feature expression by self-attention enhancement channel by channel and take into account the constraints of semantic categories to learn a better semantic segmentation model for urban street scenes. The experiments are performed on the Semantic3D dataset and our own collected vehicle Mobile Laser Scanning (MLS) point cloud dataset. The segmentation results show that the GFSAE and the WSM proposed by us can improve the semantic segmentation of point clouds in large-scale urban street scenes and prove the effectiveness of our model compared with other state-of-the-art methods.

Accurate and efficient extraction of road marking plays an important role in road transportation engineering, automotive vision, and automatic driving. In this article, we proposed a dense feature pyramid network (DFPN)-based deep learning model, by considering the particularity and complexity of road marking. The DFPN concatenated its shallow feature channels with deep feature channels so that the shallow feature maps with high resolution and abundant image details can utilize the deep features. Thus, the DFPN can learn hierarchical deep detailed features. The designed deep learning model was trained end to end for road marking instance extraction with mobile laser scanning (MLS) point clouds. Then, we introduced the focal loss function into the optimization of deep learning model in road marking segmentation part, to pay more attention to the hard-classified samples with a large extent of background. In the experiments, our method can achieve better results than state-of-the-art methods on instance segmentation of road markings, which illustrated the advantage of the proposed method.

The incidence of cardiovascular disease is increasing year by year and is showing a younger trend. At the same time, existing medical resources are tight. The automatic detection of ECG signals becomes increasingly necessary. This paper proposes an automatic classification of ECG signals based on a dilated causal convolutional neural network. To solve the problem that the recurrent neural network framework network cannot be accelerated by hardware equipment, the dilated causal convolutional neural network is adopted. Given the features of the same input and output time steps of the recurrent neural network and the nondisclosure of future information, the network is constructed with fully convolutional networks and causal convolution. To reduce the network depth and prevent gradient explosion or gradient disappearance, the dilated factor is introduced into the model, and the residual blocks are introduced into the model according to the shortcut connection idea. The effectiveness of the algorithm is verified in the MIT-BIH Atrial Fibrillation Database (MIT-BIH AFDB). In the experiment of the MIT-BIH AFDB database, the classification accuracy rate is 98.65%.

• An improved convolutional module MCB is proposed. • A hybrid down-sampling module HDSB is proposed to reduce the feature loss. • The Skip Connection+ structure is proposed to capture more context information. Background and objective: Medical image segmentation plays an important role in clinic. Recently, with the development of deep learning, many convolutional neural network (CNN)-based medical image segmentation algorithms have been proposed. Among them, U-Net is one of the most famous networks. However, the standard convolutional layers used by U-Net limit its capability to capture abundant features. Additionally, the consecutive maximum pooling operations in U-Net cause certain features to be lost. This paper aims to improve the feature extraction capability of U-Net and reduce the feature loss during the segmentation process. Meanwhile, the paper also focuses on improving the versatility of the proposed segmentation model. Methods: Firstly, in order to enable the model to capture richer features, we have proposed a novel multiscale convolutional block (MCB). MCB adopts a wider and deeper structure, which can be applied to different types of segmentation tasks. Secondly, a hybrid down-sampling block (HDSB) has been proposed to reduce the feature loss via replacing the maximum pooling layer. Thirdly, we have proposed a context module (CIF) based on atrous convolution and SKNet to extract sufficient context information. Finally, we combined the CIF module with Skip Connection of U-Net, and further proposed the Skip Connection+ structure. Results: We name the proposed network MHSU-Net. MHSU-Net has been evaluated on three different datasets, including lung, cell contour, and pancreas. Experimental results demonstrate that MHSU-Net outperforms U-Net and other state-of-the-art models under various evaluation metrics, and owns greater potential in clinical applications. Conclusions: The proposed modules can greatly improve the feature extraction capability of the segmentation model and effectively reduce the feature loss during the segmentation process. MHSU-Net can also be applied to different types of medical image segmentation tasks.

Road markings are one of the most important safety elements in a road network, and they play a critical role in traffic safety. However, the automatic extraction of road markings remains a technical challenge in the fields of smart city construction and automatic driving. This paper presents an image-translation-based method of obtaining the 3D vectors of typical road markings from mobile laser point clouds. First, ground roughness is used as a criterion to extract ground points based on the topological relationship of adjacent scan lines, and the feature images of a road surface are generated using the adapted inverse distance weighted method. Second, by comparing objective functions based on the pix2pix framework, a finely adjusted image-to-image translation model named P2P_L1 is proposed for the segmentation of road markings. The proposed model outperforms the advanced DeepLab V3+ network in terms of precision, F1-score, and mean Intersection over Union indicators in the comparative segmentation results of ten types of road markings in the Shenzhen test area. Third, methods such as node averaging and optimized iterative closest point are developed for the 3D vectorization of road markings. This study presents a new approach for the automatic extraction of road markings to provide effective technical support for the construction of smart cities.

Manhole covers, which are a key element of urban infrastructure management, have a direct impact on travel safety. At present, there is no automatic, safe, and efficient system specially used for the intelligent detection, identification, and assessment of manhole covers. In this work, we developed an automatic detection, identification, and assessment system for manhole covers. First, we developed a sequential exposure system via the addition of multiple cameras in a symmetrical arrangement to realize the joint acquisition of high-precision laser data and ultra-high-resolution ground images. Second, we proposed an improved histogram of an oriented gradient with symmetry features and a support vector machine method to detect manhole covers effectively and accurately, by using the intensity images and ground orthophotos that are derived from the laser points and images, respectively, and apply the graph segmentation and statistical analysis to achieve the detection, identification, and assessment of manhole covers. Qualitative and quantitative analyses are performed using large experimental datasets that were acquired with the modified manhole-cover detection system. The detected results yield an average accuracy of 96.18%, completeness of 94.27%, and F-measure value of 95.22% in manhole cover detection. Defective manhole-cover monitoring and manhole-cover ownership information are achieved from these detection results. The results not only provide strong support for road administration works, such as data acquisition, manhole cover inquiry and inspection, and statistical analysis of resources, but also demonstrate the feasibility and effectiveness of the proposed method, which reduces the risk involved in performing manual inspections, improves the manhole-cover detection accuracy, and serves as a powerful tool in intelligent road administration.