Snapshot imaging spectropolarimeters (ISPs) have been widely applied in the field of target detection due to their capability for real-time data acquisition. In this paper, we propose a snapshot image mapping spectropolarimeter (IMSP) based on a broadband liquid crystal polarization grating (LCPG). The system features a compact and concise optical design that enables the simultaneous acquisition of both spectral and polarization information represented by Stokes vector within a single exposure. The IMSP offers 20 spectral channels, covering a spectral range from 450 nm to 850 nm, and is capable of simultaneously measuring the s0 and s1 components of the Stokes vector. This paper presents experimental tests conducted using the IMSP system to evaluate its spectropolarimetric imaging capabilities, and provides the associated experimental results. Furthermore, a rapid calibration and image reconstruction method tailored for IMSP is developed, which effectively corrects spatial and spectral distortions while enhancing overall image quality.

In the fields of photogrammetry and computer vision, 3-D urban building model reconstruction from airborne light detection and ranging (LiDAR) point clouds has attracted significant attention in recent years. Accurately and automatically extracting local geometric structures, such as planar patches, from 3-D point cloud data directly determines the quality of subsequent 3-D model reconstruction. Considering that the roof is a crucial component of a real building, roof plane segmentation is a critical procedure in building 3-D reconstruction. In this article, a novel dual-branch transformer-based network is designed to accurately segment roof planes from airborne LiDAR point clouds. We first use PointNet++ followed with a transformer encoder to extract point-wise feature embeddings. Then, in the first branch, a transformer decoder module is applied to directly learn the instance centers of planar patches by giving a set of learned queries. Because the transformer can effectively model the relations of the queries and the global context information, the instance center positions of all planes included in the input point clouds can be accurately predicted. In this way, the number and center positions of roof planes are known before performing roof plane segmentation. In the second branch, we predict the offsets for each point using its point-wise feature to shift it toward the corresponding instance center. After that, the plane parameters for each plane instance can be estimated using the shifted points around the predicted centers, and the rest of the points are assigned to its nearest plane to generate the final roof planes. The experimental results illustrate that our approach can successfully address the plane segmentation challenge for diverse building roof structures while achieving performance superior to the current state-of-the-art techniques. We will make the source code of our approach publicly available at https://github.com/Li-Li-Whu/PlaneTransformer

This paper presents a simultaneous localization and mapping (SLAM) system to provide accurate pose estimation and dynamic scene reconstruction. Our approach proposes a Joint Point-Gaussian Splatting representation, which fully integrates the robustness of isotropic feature points in pose estimation and the flexibility of anisotropic 3D Gaussians in scene representation. This system does not need to suppress the anisotropic representation of Gaussian elements, which enables the mapping module to achieve finer scene representation with lower memory consumption. Additionally, in order to enhance the adaptability of the system in dynamic environments, we introduced a dynamic region recognition module and utilized 3D Gaussian Splatting and 4D Gaussian Splatting representations to represent static and dynamic regions respectively. Furthermore, we developed a local map management strategy for Gaussian Splatting mapping, effectively reducing the memory and computational resource usage in the mapping process. Experiments on public datasets demonstrate that our system achieves state-of-the-art tracking and mapping accuracy compared to existing baselines.

The global optimization-based color correction approach aims to minimize the color differences of multiple images by optimizing the correction model for each image. The color differences in multisource and multitemporal remote sensing images are difficult to express using a simple correction model with few parameters. When employing a more flexible correction model, the number of correction parameters and optimization equations grows rapidly with the increase in the number and resolution of input images. In addition, the correction parameters of all images are coupled together and need to be solved simultaneously. An excessive number of parameters results in solving slowly or potential failure. To solve this problem, we propose a parallelizable color correction approach that decouples the correlation of correction parameters in the optimization equations and optimizes each image separately. First, we introduce auxiliary variables that replace values related to other images in the cost function. Second, we construct optimization equations for each image and parallelly solve the correction parameters. Finally, we correct the input images through a weighted correction model to better eliminate correction artifacts. Our approach iteratively optimizes auxiliary variables and correction parameters until the correction results converge. The experimental results on several challenging datasets show that our approach significantly improves execution efficiency and obtains the global optimal solution using the flexible correction model.

Orthoimage mosaicking aims to make the seamline bypass obvious objects to produce seamless digital orthophoto maps. Nevertheless, stitching artifacts may still occur when obvious objects are inevitably crossed by the seamline or when the orthoimages are not accurately aligned geometrically. Image warping is necessary to correct these geometric misalignments. Existing image warping approaches primarily focus on natural images, with relatively few studies targeting orthoimages. Unlike natural images, orthoimages have significant geometric attributes and no longer satisfy the epipolar constraint. Therefore, most of the existing image warping methods are ineffective in dealing with geometric misalignment in orthoimage mosaicking. To solve the aforementioned problems, we propose a coarse-to-fine image warping approach using trust region optimization for orthoimage mosaicking. First, we obtain an optimal seamline and search for geometrically misaligned regions along the seamline. Second, for each region, we model the geometric alignment problem as an optimization problem of the deformation vectors for pixels. Next, the trust region method is employed to iteratively solve the optimization problem. We introduce the image pyramid strategy to achieve efficient coarse-to-fine optimization and help to avoid local optima. Finally, the orthoimages are warped based on the optimized pixel deformation vectors. However, when dealing with misaligned regions covered by multiple images, frame-to-frame warping cannot achieve optimal results. Therefore, we propose a multi-image joint optimization strategy that introduces auxiliary variables to fuse information from multiple orthoimages, ensuring seamless mosaicking. Experimental results demonstrate that, whether in two- or multi-orthoimage mosaicking, our solution has better visual and quantitative performance.

Roof plane segmentation is one of the key procedures for reconstructing three-dimensional (3D) building models at levels of detail (LoD) 2 and 3 from airborne light detection and ranging (LiDAR) point clouds. The majority of current approaches for roof plane segmentation rely on the manually designed or learned features followed by some specifically designed geometric clustering strategies. Because the learned features are more powerful than the manually designed features, the deep learning-based approaches usually perform better than the traditional approaches. However, the current deep learning-based approaches have three unsolved problems. The first is that most of them are not truly end-to-end, the plane segmentation results may be not optimal. The second is that the point feature discriminability near the edges is relatively low, leading to inaccurate planar edges. The third is that the planar geometric characteristics are not sufficiently considered to constrain the network training. To solve these issues, a novel edge-aware transformer-based network, named RoofSeg, is developed for segmenting roof planes from LiDAR point clouds in a truly end-to-end manner. In the RoofSeg, we leverage a transformer encoder-decoder-based framework to hierarchically predict the plane instance masks with the use of a set of learnable plane queries. To further improve the segmentation accuracy of edge regions, we also design an Edge-Aware Mask Module (EAMM) that sufficiently incorporates planar geometric prior of edges to enhance its discriminability for plane instance mask refinement. In addition, we propose an adaptive weighting strategy in the mask loss to reduce the influence of misclassified points, and also propose a new plane geometric loss to constrain the network training.

Roof plane segmentation from airborne light detection and ranging (LiDAR) point clouds is an important technology for three-dimensional (3D) building model reconstruction. One of the key issues of plane segmentation is how to design powerful features that can exactly distinguish adjacent planar patches. The quality of point feature directly determines the accuracy of roof plane segmentation. Most of existing approaches use handcrafted features, such as point-to-plane distance, normal vector, etc., to extract roof planes. However, the abilities of these features are relatively low, especially in boundary areas. To solve this problem, we propose a boundary-aware point clustering approach in Euclidean and embedding spaces constructed by a multi-task deep network for roof plane segmentation. We design a three-branch multi-task network to predict semantic labels, point offsets and extract deep embedding features. In the first branch, we classify the input data as non-roof, boundary and plane points. In the second branch, we predict point offsets for shifting each point towards its respective instance center. In the third branch, we constrain that points of the same plane instance should have the similar embeddings. We aim to ensure that points of the same plane instance are close as much as possible in both Euclidean and embedding spaces. However, although deep network has strong feature representative ability, it is still hard to accurately distinguish points near the plane instance boundary. Therefore, we first robustly group plane points into many clusters in Euclidean and embedding spaces to find candidate planes. Then, we assign the rest boundary points to their closest clusters to generate the final complete roof planes. In this way, we can effectively reduce the influence of unreliable boundary points. In addition, to train the network and evaluate the performance of our approach, we prepare a synthetic dataset and two real datasets. The experiments conducted on synthetic and real datasets show that the proposed approach significantly outperforms the existing state-of-the-art approaches in both qualitative evaluation and quantitative metrics. To facilitate future research, we will make datasets and source code of our approach publicly available at https://github.com/Li-Li-Whu/DeepRoofPlane.

Recovering the wavelengths from disordered speckle patterns has emerged as an exciting prospect as a wavelength measurement method due to its combination of high resolution and straightforward design. In previous studies, panel cameras were commonly utilized as the speckle image receiver. However, high cost (especially in a near-infrared range), bulky size, and low speed have limited its application in optical communications, metrology, and optical sensing. In this work, we effectively compressed speckle patterns into four intensities by using a quadrant detector (QD), bypassing the need for millions of pixels as in conventional cameras. Remarkably, wavelengths can still be recovered through only these four pixels. A new CNN based demodulation algorithm, shallow residual network (SRN), was proposed to recognize the wavelengths from the highly compressed speckle images. Finally, a wavelength precision of 4 fm (∼ 0.5 MHz) with an updating speed of ∼ 1 kHz was achieved in the demonstrations. In addition, the SRN shows a broad measurement range and good noise robustness. Compared with a camera-based system, the QD detection scheme associated with the CNN algorithm provides a compact, high-speed, and low-cost method to examine the speckle patterns, which opens new routes in many other fields.

A multifunctional surface-enhanced Raman scattering (SERS) platform integrating sensitive detection and drug resistance analysis was developed for Gram-positive bacteria. The substrate was based on self-assembled Ti3C2Tx@Au NPs films and capture molecule phytic acid (IP6) to achieve specific capture of Gram-positive bacteria and different bacteria were analyzed by fingerprint signal. It had advantages of good stability and homogeneity (RSD = 8.88

Structure from motion (SfM) recovers scene structures and camera poses based on feature matching, and faces challenges from ambiguous scenes. There are a large number of ambiguous scenes in real environment, which contain many duplicate structures and textures. The ambiguity leads to incorrect feature matches between images with similar appearance, and makes geometric misalignment in SfM. To address this problem, recent methods have focused on investigating the inconsistencies in feature topology among multi-view images. However, the feature topology is directly derived from 2D images. Thus, it is susceptible to feature occlusion caused by changes in perspective. Therefore, we propose a new method that disambiguates scenes using pose consistency rather than feature consistency. The pose consistency is conducted in 3D geometric space which is less sensitive to feature occlusion. Thus, the pose consistency is more robust than feature consistency. Our core motivation lies that the incorrect matches between ambiguous images will cause pose deviation from the global poses generated by correct matches. To detect this pose deviation, we first combine local and global information of the scene to generate the global reliable camera poses. The local information of each image is obtained by image clustering, and it strengthens the global information that is represented as the verified maximum spanning tree of clusters. Then, the global poses serve as the reference for further pose consistency verification. The global poses also enable us to perform both rotation and translation consistency verification for uncertain matches. During the pose consistency verification, the pose deviation calculated on image-level may be too small to be noticed. Thus, we propose to perform pose consistency verification at cluster-level instead of image-level to amplify the pose deviation. In the experiments, we compared our approach with several state-of-the-art methods, including COLMAP, Geodesic-SfM and TC-SfM, on both ambiguous and regular datasets. The results demonstrate that our approach achieves the best robustness, only our approach succeeds on all ambiguous image sequences (14/14). The quantitative evaluation results on image sequences with ground truth also show that our approach achieves the best accuracy (average RMSE of translation = 0.109, average RMSE of rotation = 0.827) among all methods. The source code of our approach is publicly available at https://github.com/gongyeted/MA-SfM.

Drug -resistant bacteria is highly infectious and has been appearing in food in recent years threatening human health. Therefore, effective detection of drug -resistant bacteria is crucial. In this paper, an interference -free surface -enhanced Raman scattering (SERS) platform was developed for the detection of extended -spectrum beta-lactamase E. coil (ESBL-E. coil). 4-mercaptomethylboronic acid (4-MPBA) was used as a capture molecule for drug -resistant bacteria, and self -assembled gold nanoparticles (Au NPs) were utilized to develop a highperformance SERS analytical platform for the detection of drug -resistant bacteria. Meanwhile, in order to obtain a lower detection limit and higher precision, Ti3C2Tx, modified with 4-mercaptobenzonitrile (4-MBN) and Au NPs, was used for signal enhancement and indirect detection. The SERS platform detected signals from the Raman silent region, where there was no background signal from biological sources, eliminating the interference, a detection limit was 10 CFU/mL, with a wide dynamic linear range from 10-108 CFU/mL. Meanwhile, the stability, homogeneity and reproducibility were excellent. The precise quantification capability of the SERS platform was validated by the detection of ESBL-E. coil in milk. Therefore, the method holds great promise for the development of a stable and sensitive SERS platform for the detection of drug -resistant bacteria.

In the design of the Computed Tomographic Imaging Spectrometer (CTIS), in order to optimize the holographic grating and achieve better design performance, this paper proposes a novel optimization algorithm based on the Gerchberg-Saxton (GS) iterative algorithm. This algorithm combines the weighted GS algorithm with the interior point method (IPM). By introducing weight factors for phase and amplitude in the optimization process of the GS algorithm, and incorporating the actual diffraction characteristics of the holographic grating obtained from the Computer Simulation Technology Studio Suite into the IPM optimization process, a more optimized design performance is achieved. Using this algorithm, a metasurface holographic grating is designed, which can transform the input parallel light into a dispersion image of 25 diffraction orders on a focal plane array. The transmission efficiency exceeds 72%, and the root mean square error between different diffraction orders is less than 0.1. Among them, the optimization time is shortened by approximately 70% due to a significant reduction in the number of independent variables through symmetry. Through comparison, this method can further improve the uniformity of energy distribution based on the original algorithm, avoid being trapped in local extreme values, and thus enhance the overall design quality of the CTIS.

The imaging quality of the Mapping Imaging Spectrometer (IMS) is crucial for spectral identification and detection performance. In IMS, the image mapper significantly influences the imaging quality. Traditional image mappers utilize a single-point diamond machining process. This process leads to inevitable edge eating phenomena that further results in noticeable deficiencies in imaging, impacting spectral detection performance. Therefore, we propose a manufacturing process for the image mapper based on ultra-thin layered glass. This process involves precision polishing of ultra-thin glass with two-dimensional angles, systematically assembling it into an image mapper. The surface roughness after coating is generally superior to 10 nm, with a maximum angle deviation of less than 3 '. This results in high mapping quality. Subsequently, a principle verification experimental system was established to conduct imaging tests on real targets. The reconstructed spectrum demonstrates excellent alignment with the results obtained from the Computed Tomography Imaging Spectrometer (CTIS). We thereby validate that this approach effectively resolves the issues associated with edge eating (caused by traditional single-point diamond machining), and leads to improved imaging quality. Also when compared to other techniques (like two-photon polymerization (2PP)), this process demonstrates notable advantages such as simplicity, efficiency, low processing costs, high fault tolerance, and stability, showcasing its potential for practical applications.

The computed tomography imaging spectrometer (CTIS) is a snapshot imaging spectrometer, excelling in dynamic detection tasks. It can capture two-dimensional spatial information and spectrally compressed information of a target within a single exposure time. However, traditional CTIS image reconstruction algorithms suffer from missing-cone problem, which reduces the accuracy of spectral reconstruction. In recent years, deep learning has been applied to CTIS spectral image reconstruction, significantly improving spectral reconstruction accuracy compared to traditional algorithms. However, due to the missing-cone problem, it is difficult to accurately recover the truth of spectral data cube in the real scene. Currently, most CTIS neural network reconstruction models are trained using simulated datasets of spectral data cubes and diffractive images. Because these data differ significantly from real data under actual application conditions, the established models may not be effectively applicable to real-world scenes. Therefore, we propose a new CTIS system based on slit-scanning architecture utilizing an adjustable slit aperture to obtain the real spectral data cube of the target while maintaining the simplicity of the CTIS structure. By limiting the field of view (FOV) through the slit, the area of diffraction overlap can be reduced, thereby enhancing the accuracy of CTIS spectral reconstruction using the expectation-maximization (EM) algorithm. This architecture allows us to obtain accurate spectral cubes that match the CTIS diffractive image of real-world scenes, providing a real dataset for training the reconstruction network. A prototype has been built to demonstrate the feasibility of our proposed solution. Furthermore, we also constructed a residual network based on multi-scale and attention mechanism. This network is trained using a combination of simulated and real spectral imaging data. Compared to the reconstruction performance of the EM algorithm and convolutional neural networks, our approach demonstrates superior spectral reconstruction accuracy, validating the importance of real spectral data in CTIS spectral reconstruction tasks.

The rapid development of augmented reality (AR), 3D reconstruction, simultaneous localization and mapping (SLAM), and autonomous driving requires off-the-shelf camera calibration solutions that are adaptable to cameras of different configurations in different complex scenarios. To this end, we propose a generic, robust, and accurate camera calibration framework, called Meta-Calib, by using single or multiple novel designed ArUco-encoded meta-board(s), which is dedicated to estimate accurate camera intrinsic parameters and extrinsic transformations of different multi-camera configurations. The ArUco calibration board has been redesigned to facilitate learning-based robust detection and obtain higher precision control point coordinates, which is termed the meta-board. This completely replaces the widely-used chessboard based on the corner extraction scheme to greatly alleviate the impact of image distortion on control points, especially when it is located at the boundary area of the fish-eye camera. A robust two-stage deep learning detection strategy is applied to reliably localize the ArUco-encoded inner coding region of the meta-board followed by identifying two categories of circular shapes representing “0” and “1” encoded in the ArUco pattern for decoding and orientation determination. The center points of circular shapes on the meta-board in the distorted image taken under the perspective view can be approximated through elliptical fitting with contour edges. The deviation between the fitting center points and ground-truth can be greatly suppressed when the refined sub-pixel contour edges extracted on the original image are projected to the orthographic projection view based on the camera intrinsic parameters, distortion coefficients and the prior information of the meta-board. Based on this observation, we propose a systematic iterative refinement approach to achieve the high-precision intrinsic calibration of a camera. This process involves improving the estimation of camera intrinsic parameters and fitting the center control points of circular shapes on the meta-boards in an iterative manner. The progressive nature of our approach permits reliably calibrate large distortion camera models under the presence of noisy measurements, which ensures good convergence. In addition, we also propose a graph-based multi-camera extrinsic calibration method via the corrected control points to reliably estimate both the relative poses of the meta-boards and cameras in the multi-camera system. The proposed method is not constrained by the number of cameras and meta-boards used, which makes our strategy accessible even with inflexible computer vision experts. Furthermore, we have derived the mathematical form for computing the covariance of the extrinsic transformation, which makes it possible to evaluate the uncertainty of the calibration results. Extensive experiments on a large number of both real and synthetic datasets, including perspective, fish-eye, and multiple overlapping cameras, are performed to prove the effectiveness and robustness of the developed Meta-Calib calibration framework.

Recent years, two-dimensional transition metal carbonitrides (MXene) have attracted much attention in the field of surface-enhanced Raman scattering (SERS). However, the relatively low enhancement of MXene is a major challenge. Herein, Nb2C-Au NPs nanocomposites were prepared by electrostatic self-assembly method, which have a synergistically conjugated SERS effect. The EM hot spots of Nb2C-Au NPs are significantly enlarged and expanded, while the surface Fermi level is decreased. This synergistic effect could improve the SERS performance of the system. Consequently, for the dye molecules CV and MeB, the detection limits reach 10-10 M and 10-9 M, respectively, while for biomolecule adenine, the detection limit is as low as 5 × 10-8 M. The results also show the good concentration-dependent linearity, uniformity, reproducibility and stability of SERS substrate. Nb2C-Au NPs could be a fast, sensitive and stable SERS platform for label-free and non-destructive detection. This work may expand the application of MXene based materials in the field of SERS.

Geo-localization has been widely applied as an important technique to get the longitude and latitude for unmanned aerial vehicle (UAV) navigation in outdoor flight. Due to the possible interference and blocking of GPS signals, the method based on image retrieval, which is less likely to be interfered with, has received extensive attention in recent years. The geo-localization of UAVs and satellites can be achieved by querying pre-obtained satellite images with GPS-tagged and drone images from different perspectives. In this paper, an image transformation technique is used to extract cross-view geo-localization information from UAVs and satellites. A single-stage training method in UAV and satellite geo-localization is first proposed, which simultaneously realizes cross-view feature extraction and image retrieval, and achieves higher accuracy than existing multi-stage training techniques. A novel piecewise soft-margin triplet loss function is designed to avoid model parameters being trapped in suboptimal sets caused by the lack of constraint on positive and negative samples. The results illustrate that the proposed loss function enhances image retrieval accuracy and realizes a better convergence. Moreover, a data augmentation method for satellite images is proposed to overcome the disproportionate numbers of image samples. On the benchmark University-1652, the proposed method achieves the state-of-the-art result with a 6.67% improvement in recall rate (R@1) and 6.13% in average precision (AP). All codes will be publicized to promote reproducibility.

A simple quasi-distributed fiber sensing interrogation system based on random speckles is proposed for weak fiber Bragg gratings[WFBGs]in this work.Without using tunable lasers or spectrometers,a piece of multimode fiber is applied to interrogate the WFBGs relying on the wavelength sensitivity of speckles.Instead of the CCD sensor,an InGaAs quadrant detector serves as the receiver to capture the fast-changing speckle patterns.A supervised deep learning algorithm of the multilayer perceptron architecture is implemented to process speckle data and to interrogate temperature changes or dynamic strains.The proposed demodulation system is experimentally demonstrated for WFBGs with 0.1％reflectivity.The experimental results demonstrate that the new system is capable of measuring temperature change with an accuracy of 1℃ and achieving dynamic frequency of 100 Hz.This speckle-based interrogation system paves a new way for distributed WFBGs sensing with a simple design.

Computed Tomographic Imaging Spectrometer (CTIS) has the advantages of both computed tomography technology and snapshot spectroscopy and has potential applications in many fields. However, the development of CTIS has been limited by the volume issues of key components. In consideration of the compact structure of the metamaterial, an all-dielectric metamaterial array is studied as an alternative to optimize the structure of CTIS. The array can be used for snapshot-type optical imaging, with a wide working range of 550 nm similar to 1000 nm. At the same time, the system design and simulation technology are studied by using the array, which provides a useful reference for the development of micro imaging spectrometer based on metamaterial.