Automated defect detection is an important part of manufacturing, where deep learning-based detection methods are widely used. However, these methods are often limited by the defective features in 2D images, and it is difficult to obtain significant defect features under single illumination, especially for metal parts. For the purpose of solving these problems, we propose a new defect detection framework which combines photometric stereo and object detection technology. The well-performing photometric stereo method is used to predict the surface normal, which is beneficial for defect detection since surface defects are more sensitive to gradient variations. We also construct a defect dataset with 2D original images and normal maps to establish a link between photometric stereo and object detection, which is sufficient for training and testing of deep learning models. Finally, the normal is fed into the detection model to achieve the classification and localization of defects based on the object detection algorithm. The experimental results reveal that the proposed method is able to accomplish high detection accuracy and outperforms the 2D image-based methods.

Automated inspection technology based on computer vision is now widely used in the manufacturing industry with high speed and accuracy. However, metal parts always appear in high gloss or shadow on the surface, resulting in the overexposure of the captured images. It is necessary to adjust the light direction and view to keep defects out of overexposure and shadow areas. However, it is too tedious to adjust the position of the light direction and view the variety of parts' geometries. To address this problem, we design a photometric-stereo-based defect detection system (PSBDDS), which combines the photometric stereo with defect detection to eliminate the interference of highlights and shadows. Based on the PSBDDS, we introduce a photometric-stereo-based defect detection framework, which takes images captured in multiple directional lights as input and obtains the normal map through the photometric stereo model. Then, the detection model uses the normal map as input to locate and classify defects. Existing learning-based photometric stereo methods and defect detection methods have achieved good performance in their respective fields. However, photometric stereo datasets and defect detection datasets are not sufficient for training and testing photometric-stereo-based defect detection methods, thus we create a photometric stereo defect detection (PSDD) dataset using our PSBDDS to eliminate gaps between learning-based photometric stereo and defect detection methods. Furthermore, experimental results prove the effectiveness of the proposed PSBBD and PSDD dataset.

The surfaces of real objects can visually appear to be glossy, matte, or anywhere in between, but essentially, they display varying degrees of diffuse and specular reflectance. Diffuse and specular reflectance provides different clues for light estimation. However, few methods simultaneously consider the contributions of diffuse and specular reflectance for light estimation. To this end, we propose ReDDLE-Net, which performs Reflectance Decomposition for Directional Light Estimation. The primary idea is to take advantage of diffuse and specular clues and adaptively balance the contributions of estimated diffuse and specular components for light estimation. Our method achieves a superior performance advantage over state-of-the-art directional light estimation methods on the DiLiGenT benchmark. Meanwhile, the proposed ReDDLE-Net can be combined with existing calibrated photometric stereo methods to handle uncalibrated photometric stereo tasks and achieve state-of-the-art performance.

Photometric stereo provides an important method for high-fidelity 3D reconstruction based on multiple intensity images captured under different illumination directions. In this paper, we present a complete framework, including a multilight source illumination and acquisition hardware system and a two-stage convolutional neural network (CNN) architecture, to construct inter- and intraframe representations for accurate normal estimation of non-Lambertian objects. We experimentally investigate numerous network design alternatives for identifying the optimal scheme to deploy inter- and intraframe feature extraction modules for the photometric stereo problem. Moreover, we propose utilizing the easily obtained object mask to eliminate adverse interference from invalid background regions in intraframe spatial convolutions, thus effectively improving the accuracy of normal estimation for surfaces made of dark materials or with cast shadows. Experimental results demonstrate that the proposed masked two-stage photometric stereo CNN model (MT-PS-CNN) performs favourably against state-of-the-art photometric stereo techniques in terms of both accuracy and efficiency. In addition, the proposed method is capable of predicting accurate and rich surface normal details for non-Lambertian objects of complex geometry and performs stably given inputs captured in both sparse and dense lighting distributions.

High-quality reconstruction of polished surfaces is a promising yet challenging task in the industrial field. Due to its extreme reflective properties, state-of-the-art methods have not achieved a satisfying trade-off between retaining texture and removing the effects of specular outliers. In this paper, we propose a learning based pixel-level photometric stereo method to estimate the surface normal. A feature fusion convolutional neural network is used to extract the features from the normal map solved by the least square method and from the original images respectively, and combine them to regress the normal map. The proposed network outperforms the state-of-the-art methods on the DiLiGenT benchmark dataset. Meanwhile, we use the polished rail welding surface to verify the generalization of our method. To fit the complex geometry of the rails, we design a flexible photometric stereo information collection hardware with multi-angle lights and multi-view cameras, which can collect the light and shade information of the rail surface for photometric stereo. The experimental results indicate that the proposed method is able to reconstruct the normal of the polished surface at the pixel level with abundant texture information.

In this paper, we aim to improve the performance of single-image super-resolution (SISR) by designing a more effective feature extraction module and a better fusion scheme for integrating hierarchical features. Firstly, we propose a selective multi-scale module (SMsM) to adaptively aggregate multi-scale features via self-learned weights and thus extract more distinctive representation. Then, we design an attentive global feature fusion (AGFF) scheme to reduce the redundant information inside the extracted hierarchical features by employing a gate mechanism (in the form of group convolution) and adaptively re-calibrate the features with channel-wise attention weights before fusion. Stacked SMsMs and AGFF compose a novel network which is termed selective multi-scale network (SMsN). Extensive experimental results demonstrate that our SMsN model outperforms some state-of-the-art SISR methods in terms of accuracy and efficiency.

The agricultural photovoltaic panel integrated greenhouse (PVIG) are attempted to improve the utilization of solar energy for sustainable agriculture. This study proposes a numerical method to predict the radiation distribution and the electricity yield of a PVIG throughout the year using 3D radiation model and thermal model. The radiation model is validated in a PVIG in the southeast of China with 36 rooftop PV panels of straight-line layout and the PV array covering 25% of the greenhouse roof area. According to simulation of conventional greenhouse and PVIG with straight-line (La), crisscross (Lb) and checkerboard (Lc) PV panels’ layouts, the performance of different PVIG is evaluated in terms of the non-uniformity of the radiation distribution, the radiation intensity and the electricity yield of PV panels. The non-uniformity of the hourly radiation under the La layout is not much different from the layout Lb and Lc, and the cumulative weekly radiation under the layout of La is increased by 8.3% and 12.7% in the experimental PVIG, respectively. The thermal model of PV panels mounted on the greenhouse roof is developed to accurately predict the PV electricity yield, considering the impact of greenhouses and external environments on PV panels. In contrast to the layouts of La and Lb, the annual electricity yield of PV panels with the layout of Lc raises by 3.2 and 2.2 kWh m−2 respectively. Compared with conventional greenhouse, the distribution and intensity of cumulative weekly radiation are not significantly affected by the optimal PV array covering 25% of the greenhouse roof area. These results indicate that the 3D radiation model and thermal model can be used as an efficient tool to optimize the performance of PVIG.