In endoscopic surgery, reconstructing dynamic scenes using 4D Gaussian Splatting (4DGS) offers high-quality results but suffers from severe geometric instability under sparse-view conditions. This instability stems from geometrically inconsistent priors and the lack of a synergistic regularization scheme. To address these root causes, we introduce a novel framework built upon two complementary modules. First, a Depth Optimization framework produces a scale-consistent geometric prior by fusing multi-resolution depth maps and anchoring the scale via sparse structure-from-motion (SfM) points. Second, a Multi-faceted Geometric Regularization scheme employs a trio of complementary constraints (across global-local depth, virtual views, and surface normals) that work in synergy to mitigate severe overfitting. Extensive experiments on two public surgical datasets demonstrate superior sparse-view reconstruction performance over representative baselines at real-time rendering speeds. This work delivers a robust and efficient solution for high-fidelity 4D scene reconstruction in minimally invasive endoscopy.

The integration of laser vision sensors in robotic welding improves seam tracking accuracy, but welding noise poses significant challenges. Our research introduces WeldNet, enhances laser stripe extraction, significantly outperforming traditional and deep neural network (DNN) solutions in efficiency and measurement precision. WeldNet comprises lightweight modules for optimal feature extraction, including Multi-Part Channel Convolution (MPC) blocks, Parallel Shift Multilayer Perceptrons (PS-MLP), and Serial Shift MLP (SS-MLP). A specially designed data augmentation strategy is also integrated to address the complex noise encountered in robotic welding. Experimental results demonstrate WeldNet’s effectiveness in reducing welding noise interference, achieving a real-time processing speed of 145 FPS on RTX 2080 Ti GPU, approximately 5x faster than existing state-of-the-art methods. With a Dice coefficient of 87.52% and an IoU value of 77.82%, WeldNet not only enhances operational efficiency but also markedly improves precision in industrial robotic welding.

Brain tumor segmentation methods based on multi-modal MRI perform significantly well when data is complete. In clinical settings, the absence of modalities due to artifacts and equipment problems often renders these methods ineffective. Current research attempts to train a universal model to adapt to 15 different random combinations of missing modalities. However, due to the random and complex nature of missing modality combinations across different cases, a single model faces challenges in dynamically adjusting its processing strategy to accommodate specific missing modality scenarios, ultimately leading to diminished segmentation accuracy. In this work, we introduce an end-to-end Incomplete Multi-modal Diffusion Brain Tumor Segmentation (IMD-TumorSeg) framework, which is designed to handle various scenarios with missing modalities. Specifically, the model incorporates independent generative modules for each modality and introduces an adaptive conditional integration mechanism to dynamically adjust the weight fusion between missing and available modalities. In addition, an attention-driven diffusion strategy is proposed to facilitate collaborative learning between the diffusion process and the segmentation network. Furthermore, by integrating an image estimator, the framework evaluates the similarity between generated and real images in real-time, optimizing the generation process and ensuring both visual and semantic consistency of the generated images. Extensive experimental results on the BraTS 2018 and BraTS 2020 datasets demonstrate that IMD-TumorSeg exhibits superior performance and effectiveness in handling missing modalities compared to state-of-the-art methods.

Multi-modal medical image translation and segmentation are essential for achieving accurate diagnosis and treatment. However, existing methods often suffer from semantic shifts during modality translation, leading to issues such as vascular discontinuity and anatomical deformation, which further degrade downstream segmentation performance. To address these challenges, we propose a Semantic-Robust multimodal medical image Translation and Segmentation framework (SR-MedTS). In this respect, we design an end-to-end dualstream architecture composed of a generator, discriminator and semantic robustness module. Moreover, we propose a channelspatial attention mechanism embedded in the skip connections between the encoder and decoder of the segmentation generator network to enhance boundary recognition. Finally, a multiloss function is defined to optimize the overall architecture. Extensive experiments on a multi-modal abdominal dataset and a six-site prostate dataset demonstrate that SR-MedTS significantly improves cross-modal segmentation performance using only 10 % of annotated data, showing strong potential in lowresource medical imaging scenarios. Our code is available at https://github.com/susu337/SR-MedTS.

Deep networks have made remarkable progress in Multi-View Stereo (MVS) task in recent years. However, the problem of finding accurate correspondences across different views under ill-posed matching situations remains unresolved and crucial. To address this issue, this paper proposes a Geometry-enhanced Attentive Multi-View Stereo (GA-MVS) network, which can access multi-view consistent feature representation and achieve accurate depth estimation in challenging situations. Specifically, we propose a geometry-enhanced feature extractor to explore illumination-invariant geometric features and incorporate them with common texture features to improve matching accuracy when dealing with view-dependent photometric effects, such as shadow and specularity. Then, we design a novel attentive learning framework to explore per-pixel adaptive supervision, effectively improving the depth estimation performance of textureless regions. The experimental results on the DTU and Tanks & Temples benchmarks demonstrate that our method achieves state-of-the-art results compared to other advanced MVS models.

Microaneurysms (MAs) in fundus images are a crucial indicator for the early detection of diabetes and the treatment of diabetic retinopathy. In this paper, we propose an efficient method for microaneurysms detection. We propose an Att-UCtransnet network to extract the MA candidate region, where only positive samples which contain at least one microaneurysm are chosen. Inspired by the concept of the effective receptive field (ERF), we design an algorithm to adaptively select the appropriate patch size, preserving crucial information around the microaneurysm while minimizing interference from irrelevant environments. Besides, we introduce a classification network for accurate mass detection in whole fundus images. Extensive experiments conducted on four publicly available datasets demonstrate that our method outperforms the state-of-the-art approaches in MA detection.

Though deep learning-based surgical smoke removal methods have shown significant improvements in effectiveness and efficiency, the lack of paired smoke and smoke-free images in real surgical scenarios limits the performance of these methods. Therefore, methods that can achieve good generalization performance without paired in-vivo data are in high demand. In this work, we propose a smoke veil prior regularized two-stage smoke removal framework based on the physical model of smoke image formation. More precisely, in the first stage, we leverage a reconstruction loss, a consistency loss and a smoke veil prior-based regularization term to perform fully supervised training on a synthetic paired image dataset. Then a self-supervised training stage is deployed on the real smoke images, where only the consistency loss and the smoke veil prior-based loss are minimized. Experiments show that the proposed method outperforms the state-of-the-art ones on synthetic dataset. The average PSNR, SSIM and RMSE values are 21.99±2.34, 0.9001±0.0252 and 0.2151±0.0643, respectively. The qualitative visual inspection on real dataset further demonstrates the effectiveness of the proposed method.

In this study, we present a novel approach to enhancing video anomaly detection by integrating an Adaptive Prototypical Network (APN) with an Enhanced Meta-Prototypical Network (EMPN) within a 3D-convolutional neural network (CNN) encoder-decoder architecture. Our method addresses the limitations of traditional anomaly detection methods by effectively capturing the intricate spatio-temporal dynamics inherent in video data. By dynamically updating representations of normal states, our model facilitates rapid adaptation to new and unseen scenes with minimal computational overhead. Furthermore, we introduce a post-processing step utilizing a median filter to refine anomaly scores, reducing noise and false positives. Extensive experiments conducted on various benchmark datasets demonstrate that our approach outperforms state-of-the-art methods on two datasets, showcasing its superior performance.

The image noise generated by the welding process, such as arc light, splash, and smoke, brings significant challenges for the laser vision sensor-based welding robot to locate the weld seam and accurately conduct automatic welding. Currently, deep learning-based approaches surpass traditional methods in flexibility and robustness. However, their significant computational cost leads to a mismatch with the real-time requirement of automated welding. In this paper, we propose an efficient hybrid architecture of Convolutional Neural Network (CNN) and transformer, referred to as Dynamic Squeeze Network (DSNet), for real-time weld seam segmentation. More precisely, a lightweight segmentation framework is developed to fully leverage the advantages of the transformer structure without significantly increasing computational overhead. In this respect, an efficient encoder, which aims to increase its features diversity, has been designed and resulted in substantial improvement of encoding performance. Moreover, we propose a plug-and-play lightweight attention module that generates more effective attention weights by exploiting statistical information of weld seam data and introducing linear priors. Extensive experiments on weld seam images using NVIDIA GTX 1050Ti show that our approach reduces the number of parameters by 54x, decreases computational complexity by 34x, and improves inference speed by 33x compared to the baseline method TransUNet. DSNet achieves superior accuracy (78.01% IoU, 87.64% Dice) and speed performance (100 FPS) with lower model complexity and computational burden than most state-of-the-art methods. The code is available at https://github.com/hackerschen/DSNet.

Monocular depth estimation is a classic research topic in computer vision. In recent years, development of Convolutional Neural Networks (CNNs) has facilitated significant breakthroughs in this field. However, there still exist two challenges: (1) The network struggles to effectively fuse edge features in the feature fusion stage, which ultimately results in the loss of structure or boundary distortion of objects in the scene. (2) Classification based studies typically depend on Transformers for global modeling, a process that often introduces substantial computational complexity overhead as described in Equation 2. In this paper, we propose two modules to address the aforementioned issues. The first module is the Boundary Attention Module (BAM), which leverages the attention mechanism to enhance the ability of the network to perceive object boundaries during the feature fusion stage. In addition, to mitigate the computational complexity overhead resulting from predicting adaptive bins, we propose a Shift Window Adaptive Bins (SWAB) module to reduce the amount of computation in global modeling. The proposed method is evaluated on three public datasets, NYU Depth V2, KITTI and SUNRGB-D, and demonstrates state-of-the-art (SOTA) performance.

During laparoscopic surgical procedures, smoke generated by sources such as lighting, gas, and machinery often obscures the surgeon’s view, degrading image quality and hindering the surgery’s progress. To address this issue, this study introduces a smoke removal framework specifically designed to enhance the clarity of laparoscopic videos. We have employed a deformable convolution module guided by a mutual attention mechanism, focused on modeling temporal information and guiding smoke removal. This module not only stably extracts temporal information from adjacent smoke frames but also enhances the representation of relevant features. Additionally, we incorporated Local Binary Patterns (LBP) as a texture prior, which maintains the semantic coherence of low-level image features and optimizes the image quality after smoke removal. Experimental results confirm that our method demonstrates exceptional performance in both simulated environments and actual surgical scenarios.

The mainstream techniques to detect microaneurysm (MA) in medical images involve the use of stacked convolutional neural networks (SCNNs). But these techniques cannot provide adequate detecting performance for the MA scenarios with limited information and different sizes. In this article, a hierarchical pyramid network with a “T” structure is proposed to detect MAs in retinal fundus images, which can overcome the difficulties caused by the above scenarios. We design a data preparation (DP) technique that can adaptively compute the best receptive fields of MAs to generate various patch sizes, resulting in multisize datasets for training. The “T” network consists of two pyramid feature extractors, rather than a deep network to adequately extract features of MAs and avoid feature loss caused by deep layers. Experiments on five public retinal datasets demonstrate that our method achieves the state-of-the-art performance.

Multi-modal MRI has become a valuable tool in medical imaging for diagnosing and investigating brain tumors, as it provides complementary information from multiple modalities. However, traditional methods for multi-modal MRI segmentation using UNet architecture typically fuse the modalities at an early or mid-stage of the network, without considering the inter-modal feature fusion or dependencies. To address this, a novel CMMFNet (cross-modal multi-scale fusion network) is proposed in this work, which explores both intra-modality and inter-modality relationships in brain tumor segmentation. The network is built on a transformer-based multi-encoder and single-decoder structure, which performs nested multi-modal fusion for high-level representations of different modalities. Additionally, the proposed CMMFNet uses a focusing mechanism that extracts larger receptive fields more effectively at the low-level scale and connects them to the decoding layer effectively. The multi-modal feature fusion module nests modality-aware feature aggregation, and the multi-modal features are better fused through long-term dependencies within each modality in the self-attention and cross-attention layers. The experiments showed that our CMMFNet outperformed state-of-the-art methods on the BraTS2020 benchmark dataset in brain tumor segmentation.

Microaneurysms (MAs) are the earliest lesions diabetic retinopathy (DR), and the accurate MA detection can assist in early diagnosis of diabetes. Many effective detection techniques have been proposed recently, the interference of blood vessels and blurry boundaries of MAs reduce the performance of these methods. To address these problems, this paper proposes an accurate MA detection method. In the candidate detection stage, a shape suppression filter is proposed to remove strip-shaped blood vessels and retain the dark blobs. In order to enhance blurry boundaries of MAs, we apply the additive bias correction level set to locate circular regions. In the next stage of feature extraction, a new set of features based on gray level co-occurrence matrix (GLCM) is extracted for each candidate to discriminate the MAs from non-MAs candidates using the random undersampling boosting (RUSBoost) classifier. We test our method on four public datasets, resulting in the optimal performance to the existing methods. It achieves average sensitivity values of 0.672, 0.721, 0.602 and 0.735, respectively.

In the context of image/video guided surgery, mixed reality (MR) is used for visualization and interaction with 3D organs models, including tumors and vascular structures. For laparoscopic liver resection, the use of MR is a very promising approach, set to revolutionize planning and execution of these complex interventions. This technology allows surgeons to collaborate on planing and executing laparoscopic liver resections, simplifying complex procedures, and enhancing patient outcomes. Furthermore, it gives more confidence to the surgeon about his performance and guides him to achieve higher accuracy in the execution of the planned resection. To achieve this, accurate organ models are created from high-quality MR/CT medical images and integrated in the MR visualization platform for planning and navigation by the surgeons. In this chapter, we will give an overview about the different steps involved in such procedures, based mainly on work we performed in Norway.

Automatic segmentation of abdominal organs in CT scans plays an important role in clinical practice. However, most existing benchmarks and datasets only focus on segmentation accuracy, while the model efficiency and its accuracy on the testing cases from different medical centers have not been evaluated. To comprehensively benchmark abdominal organ segmentation methods, we organized the first Fast and Low GPU memory Abdominal oRgan sEgmentation (FLARE) challenge, where the segmentation methods were encouraged to achieve high accuracy on the testing cases from different medical centers, fast inference speed, and low GPU memory consumption, simultaneously. The winning method surpassed the existing state-of-the-art method, achieving a 19× faster inference speed and reducing the GPU memory consumption by 60% with comparable accuracy. We provide a summary of the top methods, make their code and Docker containers publicly available, and give practical suggestions on building accurate and efficient abdominal organ segmentation models. The FLARE challenge remains open for future submissions through a live platform for benchmarking further methodology developments at https://flare.grand-challenge.org/.

Background: Learning-based methods have achieved remarkable performances on depth estimation. However, the premise of most self-learning and unsupervised learning methods is built on rigorous, geometrically-aligned stereo rectification. The performances of these methods degrade when the rectification is not accurate. Therefore, we explore an approach for unsupervised depth estimation from stereo images that can handle imperfect camera parameters. Methods: We propose an unsupervised deep convolutional network that takes rectified stereo image pairs as input and outputs corresponding dense disparity maps. First, a new vertical correction module is designed for predicting a correction map to compensate for the imperfect geometry alignment. Second, the left and right images, which are reconstructed based on the input image pair and corresponding disparities as well as the vertical correction maps, are regarded as the outputs of the generative term of the generative adversarial network (GAN). Then, the discriminator term of the GAN is used to distinguish the reconstructed images from the original inputs to force the generator to output increasingly realistic images. In addition, a residual mask is introduced to exclude pixels that conflict with the appearance of the original image in the loss calculation. Results: The proposed model is validated on the publicly available Stereo Correspondence and Reconstruction of Endoscopic Data (SCARED) dataset and the average MAE is 3.054 mm. Conclusion: Our model can effectively handle imperfect rectified stereo images for depth estimation.

With the unprecedented developments in deep learning, automatic segmentation of main abdominal organs seems to be a solved problem as state-of-the-art (SOTA) methods have achieved comparable results with inter-rater variability on many benchmark datasets. However, most of the existing abdominal datasets only contain single-center, single-phase, single-vendor, or single-disease cases, and it is unclear whether the excellent performance can generalize on diverse datasets. This paper presents a large and diverse abdominal CT organ segmentation dataset, termed AbdomenCT-1K, with more than 1000 (1K) CT scans from 12 medical centers, including multi-phase, multi-vendor, and multi-disease cases. Furthermore, we conduct a large-scale study for liver, kidney, spleen, and pancreas segmentation and reveal the unsolved segmentation problems of the SOTA methods, such as the limited generalization ability on distinct medical centers, phases, and unseen diseases. To advance the unsolved problems, we further build four organ segmentation benchmarks for fully supervised, semi-supervised, weakly supervised, and continual learning, which are currently challenging and active research topics. Accordingly, we develop a simple and effective method for each benchmark, which can be used as out-of-the-box methods and strong baselines. We believe the AbdomenCT-1K dataset will promote future in-depth research towards clinical applicable abdominal organ segmentation methods.

Image Data Part 3 of AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem Paper: https://ieeexplore.ieee.org/document/9497733/ Other two parts: Part 1, Part 2