Recent advancements in 3D reconstruction helped endoscopy doctors analyze the patients' gastrointestinal surfaces and abnormality detections. In this work, we expand this development further with a reconstruction method based on both classic techniques like structure from motion and recent advanced techniques like neural radiation fields and Gaussian splatting with new Gaussian encoding-decoding modules. In addition, an unique dataset was collected with some videos from daily endoscopy examinations. This development helped us achieve better reconstruction results and lower training time compared to existing methods.
In this paper, we address critical challenges in medical image generation using deep learning techniques. While convolutional neural networks and generative adversarial networks (GANs) have achieved remarkable results in various image generation tasks, their application to medical imaging faces unique obstacles. These include the complexity and diversity of medical images, limitations in discriminator network structures, and the risk of model collapse and gradient vanishing in multi-scale discriminators. To overcome these issues, we propose a novel deterministic guided progressive GAN that specifically targets regions of interest (ROI) in medical images. Our approach progressively integrates adversarial generative networks, evolving from single-scale to multi-scale discriminators, to produce higher quality images. We demonstrate the efficacy of our model in generating high-precision cross-modal medical images through four comprehensive evaluation criteria, providing both quantitative and qualitative evidence of its performance compared to real images. This innovative method promises to significantly advance the field of medical image generation, potentially enhancing diagnostic accuracy and research capabilities in healthcare.
Automated annotation of circuit structures can generate hierarchical representations of analog circuit networks, thereby advancing the development of automated analog circuit design tasks. This paper introduces a graph attention network-based model that transforms circuit netlists into graph structures, proposes a feature extraction strategy to learn and predict the circuit structures composed of nodes in the netlists, and presents a method for quickly generating a large number of SPICE circuit netlists to provide ample data for training the graph model. Experiments compared the recognition effects of graph convolutional networks, graph isomorphism networks, and GraphSAGE on the same dataset. The results show that the GAT model outperforms the other models in accuracy, precision, and mean average precision, achieving 90.9
Semantic segmentation plays a crucial role in computer-aided medical image analysis by achieving important and useful regions, which are vital for various diagnostic tasks. Recently, vision transformers (ViTs) have emerged as the leading approach in medical image segmentation, outperforming traditional convolutional neural networks (CNNs). The incorporation strategies of the ViTs for medical segmentation are dominated to leverage the widely used U-shape like architecture (U-Net) while replace the convolution blocks in both encoder and decoder paths using transformer blocks. It remains uncertain which components of the incorporated transformer block contribute most significantly to segmentation results in the medical field. This study presents a hierarchical feature aggregation method based on hierarchical Transformer features to enhance the performance of ViT-based architecture in data-constrained medical image segmentation. Specifically, our approach employs the hierarchical vision Transformer to configure the main encoder path for extracting multi-scale semantic features, and leverages several residual blocks to achieve local representation with detail spatial information. Then, we introduce a hierarchical feature aggregation module (HFAM) to serve as the decoder path for fusing multi-scale semantic features and residual spatial features. Compared with the existing transformer-based U-Net, the explored HFAM can not only effectively combine the diverse contexts but also potentially reduce the computational complexity. Experiments on 3 different medical image segmentation benchmarks have demonstrated our proposed method consistently outperformers the conventional U-Net, and various Transformer-based U-Net.
Urban land use maps at the building instance level are crucial geo-information for many applications, yet they are challenging to obtain. Land-use classification based on spaceborne or aerial remote sensing images has been extensively studied over the last few decades. Such classification is usually a patch-wise or pixel-wise labeling over the whole image. However, for many applications, such as urban population density estimation or urban utility mapping, a classification map based on individual buildings (residential, commercial, mixed-type, and religious) is much more informative. Nonetheless, this type of semantic classification still poses fundamental challenges, such as retrieving fine boundaries of individual buildings. Street view images (SVI) are highly suited for predicting building functions because building facades provide clear hints. Although SVIs are used in many studies, their application in generating building usage maps is limited. Furthermore, their application to Indian cities remains void. In this paper, we propose a comprehensive framework for classifying the functionality of individual buildings. Our method leverages the YOLOs model and utilizes SVIs, including those from Google Street View and OpenStreetMap. Geographic information is employed to mask individual buildings and associate them with the corresponding SVIs. We created our own dataset in Indian cities for training and evaluating our model.
Video frame interpolation aims to synthesize non-existent frames between two consecutive frames in a video, and its importance can be seen from its wide applications in computer vision. The key to video frame interpolation is predicting the intermediate motions between two given frames, so that the synthesized frames are coherent with the input video. However, the existing approaches of imposing assumptions on motions, such as linear and quadratic trajectories, are not generalizable to complex motions in real-world videos. In this work, we propose long-term propagation for robust and general motion prediction in video frame interpolation. To more thoroughly understand the motion trajectories, we propose to implicitly track the motion paths through a long sequence of video frames. The motion features are then used to refine the motion predicted from the primitive motion assumptions. Our proposed long-term propagation of motion can be easily integrated into existing video frame interpolation approaches. Quantitative and qualitative results demonstrate that long-term propagation effectively improves interpolation performance when incorporated into various state-of-the-art baselines. In addition, we present a series of analytical experiments to study the mechanism, advantages, and limitations of long-term propagation in video frame interpolation to inspire future works.
In autonomous driving, visual features play a crucial role. End-to-end driving models (E2EDMs) extract numerous visual features from the driving environment to solve driving tasks. However, these visual features are often difficult for humans to understand, leading to explainability issues. This study aims to improve the explainability of E2EDMs by enhancing their ability to extract semantically meaningful and driving-related visual features, like vehicles, pedestrians, and traffic signals. The training process of E2EDMs involves leveraging a backbone that is pre-trained on large datasets and subsequently fine-tuned for driving tasks. To address the explainability issue of E2EDMs, previous studies have designed complex E2EDMs during the fine-tuning stage. In this paper, we enhance the explainability by improving the backbone's ability to recognize driving-related features, i.e., object features. We propose CROp-based COntrastive DIscriminative LEarning (CROCODILE), an additional pre-training method for the backbone. CROCODILE improves the backbone's ability to preserve driving-related features while suppressing irrelevant features. Then, during fine-tuning, only driving-related features will be used for driving action prediction, thereby achieving high explainability. In addition, CROCODILE eliminates the need for complex structures in the fine-tuning stage.
Denoising diffusion models have recently demonstrated great success in generating detailed pixel-wise representations for image generation. Applications like Dall-E, Stable Diffusion, and Midjourney have showcased impressive image-generation capabilities, sparking significant discussion within the community. Recent studies have also highlighted the utility of these models in various other vision tasks, including image deblurring, super-resolution, and image segmentation. This work introduces a novel Adaptive Dual Attention into Diffusion model for 3D medical image segmentation. Applying diffusion models to 3D medical image segmentation presents significant challenges. The alignment of semantic features necessary for conditioning the diffusion process with noise embedding is often inadequate. Additionally, traditional U-Net backbones in diffusion models are not sufficiently sensitive to the contextual information required for accurate pixel-level segmentation during reverse diffusion. Our method, which integrates Adaptive Dual Attention into Diffusion, addresses these issues by capturing local and global contextual information, enhancing the precision and robustness of 3D image segmentation. Our approach surpasses current state-of-the-art methods on the BraTS2020 dataset, achieving higher segmentation accuracy. This improved performance can significantly aid in diagnosing and treating medical conditions by enabling highly accurate segmentation of anatomical structures in 3D medical images.
Facial expression recognition (FER) is a key component in various AI-based systems and has been extensively studied. However, most FER research has focused on clear and simple basic emotions such as happiness and sadness, which are not suitable for real-world applications where numerous many ambiguous and complex emotions exist. Complex emotions are challenging to define and require ample data for each emotion to train a FER model. Moreover, due to their ambiguous nature, these emotions are difficult to annotate. Consequently, the difficulty in constructing comprehensive databases is a significant bottleneck in recognizing complex emotions. In this study, we propose complex emotion recognition method using only a database of basic emotions based through angle-base metric learning. This approach can mitigate the reduction in recognition accuracy caused by insufficient data and allows for the definition of new emotions in the future, unlike general FER tasks that require pre-definition of emotions.
Understanding scenes requires not only the detection objects but also the recognition of the interactions between them. Human-Object Interaction (HOI) detection plays a crucial role in enhancing contextual comprehension by identifying the interactions between humans and objects, which is essential for building more robust and intelligent vision systems. While DETR-based models have shown significant success in HOI detection, they are hindered by slow training convergence. The SOV-STG method has attempted to address this challenge in previous research. To further improve the learning efficiency and accuracy of SOV-STG, we introduce a novel Separate Guided Denoising training strategy specifically designed for HOI detection. Our approach separates the denoising of noised ground truth data for both the human-object decoder and the verb decoder, enabling more efficient and targeted training. Furthermore, we enhance training performance by merging redundant human-object pair annotations, and filtering and regenerating noised bounding boxes. The proposed method was validated on the HICO-DET dataset, achieving state-of-the-art results. Our contributions include a novel training strategy that improves accuracy and ablation studies demonstrating its effectiveness.
Vector images are widely used in the design field, particularly for logos and icons, due to their scalable properties. Consequently, the flexible and high-quality creation of such images is expected to support creative activities. In this study, we leverage a recently proposed differentiable renderer and the strong raster image generation capabilities of Stable Diffusion to generate vector-format logo images. This is achieved through optimizing vector parameters based on losses calculated from text prompts and shape images. Additionally, we address the self-intersection issue, a common challenge in vector image generation through optimization methods, by introducing a new technique called Radiation Loss. This approach explicitly monitors control points to enhance the quality of the output. While this method successfully generates logo images that maintain the input text and shape, challenges remain, including the persistence of unnecessary paths and difficulty in controlling the output entirely by text prompts. The experimental results showed the effectiveness of the proposed methods.
Continuous gesture recognition is a crucial task in human-computer interaction. Unlike isolated gesture recognition, where individual gestures are analyzed independently, continuous recognition involves detecting and classifying multiple gestures seamlessly from continuous video streams. In this paper, we propose a method for continuous gesture recognition. Our proposed model operates in two stages: isolated gesture recognition and a sliding window-based approach for continuous gesture recognition. For isolated gesture recognition, we propose a dual encoder method named TDDNet, stand for Text-Enhanced DDNet, that integrates a skeleton encoder based on the DDNet model [6] with a text encoder based on CLIP. We evaluate our model on a self-collected dataset comprising 19 gestures relevant to human-COBOT interaction, collected from 50 subjects. Experimental results demonstrate that our model improves isolated gesture recognition accuracy from 84.2 https://github.com/duclvQ/improved_DDNet.
Automatic lesion segmentation in endoscopic images is crucial for mitigating the risk of omissions during analysis, particularly for inexperienced physicians or in situations of medical overload. Traditional segmentation models predominantly rely on pixel-level labeled images, often neglecting auxiliary information such as physicians' diagnostic conclusions. This study proposes a novel approach to harness available lesion information-including segmentation regions, physician conclusions, and supplementary disease descriptions-to improve segmentation efficacy. Our method builds upon the successful integration of CNN and Vision Transformer architectures from the LViT model, originally designed for lung cancer lesion segmentation from X-ray images using dual inputs: images and text. We propose a new framework, namely called LViTES with four key advancements: 1) optimizing the LViT architecture to enhance image feature extraction by incorporating the EfficientNet backbone and integrating Cross-Attention, while also reducing model complexity and parameters; 2) addressing the scarcity of textual descriptions in current datasets by developing a module that generates text from segmentation masks based on attributes like shape, location, size, and quantity; 3) incorporating both image and text inputs during training while allowing adaptive prediction with only image inputs to align with typical use cases; and 4) evaluating model performance using both generated text and physician-provided descriptions. The effectiveness of our approach is validated on three types of lesions-gastric cancer, esophageal cancer (our self-collected datasets), and polyps (Kvasir-SEG dataset)-demonstrating superior performance compared to state-of-the-art methods.
Many semantic segmentation methods employ various attention mechanisms to improve segmentation accuracy. However, as the accuracy of the model increases, the computational cost is relatively expensive, which is not favorable for some practical applications. To solve this problem, this paper presents an efficient and effective convolutional attention network (E2CANet), which is designed to achieve a good trade-off between segmentation accuracy and computational efficiency. E2CANet adopts an encoder-decoder architecture with skip connections to preserve details and semantic information. For the encoder, we use cheap convolutional operations to introduce two different attentions, i.e. global attention and multi-scale attention, which can significantly reduce the computational cost while highlighting important features and suppressing unnecessary ones. A lightweight All-MLP decoder, which only consists of six linear layers, is used to aggregate features from the encoder. The simple design of this decoder is also the key to reduce computational complexity. Extensive experiments are performed on ADE20K, Cityscapes, and COCO-stuff datasets. The proposed E2CANet delivers very competitive results on all datasets. Especially, E2CANet-Tiny (a lightweight version of E2CANet) achieves 41.92 https://github.com/muyuerong/E2CANet .
In recent years, the combined use of Vision Transformer (ViT) and Convolutional Neural Network (CNN) has shown promising results in tasks related to satellite imagery. In our study, we propose a 3D-CmT (3D-CNN meets Transformer) model for Hyperspectral Image Classification. This model leverages the unique capabilities of both 3D-CNN and ViT to effectively classify images captured by hyperspectral imaging. To learn the local features of the narrow and contiguous electromagnetic spectrum of the hyperspectral images, we utilize a 3D-CNN under the spectral feature extraction (SFE) module. Subsequently, a transformer encoder (TE) module is applied on top of the 3D-CNN to incorporate global attention and model long-range dependencies for spatial information in the images. We conducted experiments using commonly used hyperspectral image datasets and performed various ablation studies, such as evaluating the impact of image patch size and different percentages of training samples. The performance of our proposed model is comparable to that of other CNN-based, transformer-based, and hybrid CNN-Transformer-based models in terms of model parameters and accuracy. In addition, we conducted quantitative and qualitative analyses to assess the performance of our model.
Image generation in 2D and 3D has become an active research topic in Deep Learning. Single or multiple input images with non-orthogonal views are used for another shape and texture with different viewing angles. On the other hand, Computer-Aided Design (CAD) relies on handling-based 3D generation, i.e., isometric view images, from three orthographic view line drawings in 2D. However, since unique viewing pairs of such 2D and 3D images are required to train, SOTA models are insufficient to generate desirable images. More spatial and geometrical constraints remain undone due to less corresponding image features between images. This paper proposes IsoTGAN with a GAN-based generator with the Transformer in its generator, where three images of an object's front, side, and top view are input. The encoder is trained by the spatial and geometrical relation among three view images. A novel Gaussian Enhanced Euclidean attention mechanism and a geometry-constrained loss function are also proposed for further local image feature enhancement. Extensive experiments on the SPARE3D dataset demonstrate that the proposed IsoTGAN outperforms State-of-the-art (SOTA) models, i.e., DINO, regarding local and global image feature accuracy. This helps generate 3D isometric view images in auto-CAD system.
A crime is an unlawful action subject to punishment by a governing authority, causing harm not only to individuals but also to the well-being of a community, society, or the state. According to the latest annual report from the National Crime Records Bureau of India, there were 445,256 cases of crimes against women registered in 2022, representing a 4
This paper presents a novel training-free semantic segmentation method that leverages a pre-trained large-scale image generation model incorporating the Multi-modal Diffusion Transformer (MM-DiT) architecture. Inspired by training-free segmentation techniques using the U-Net-based noise removal model in the Stable Diffusion framework, our approach extracts cross-attention maps between textual and visual features during the inference stages of the MM-DiT to generate mask images. Experimental results demonstrate that our method achieves segmentation accuracy comparable to CLIP-based and U-Net-based stable diffusion methods. While the direct segmentation scores are relatively modest, the significance of our work lies in the exploration of cross-attention maps within the DiT. This investigation provides critical insights that could advance training-free segmentation methodologies and enhance the interpretability of diffusion-based models.
Modern surveillance societies constantly face bottlenecks due to manual monitoring of huge amounts of data generated by surveillance infrastructure. The limitation of manual monitoring is further aggravated by challenging weather conditions such as fog, rain, mist, etc. This gave rise to automated surveillance making Video anomaly detection (VAD) one of the most sought-after domains in computer vision. The availability of data that contain weather-induced variations is a key factor in the effectiveness of Data-driven approaches that rely on data for precise modeling. To this end, we have presented a brief review of previous datasets and their limitations on parameters such as size, scene variations, activities covered, effect of weather phenomena, etc. To leverage the intricate relationship between data and model we present a novel human-centric surround view dataset where each scripted activity is recorded simultaneously by 4 strategically placed cameras to capture effects of varying distance, angle, height, and illumination on the same scene. The proposed dataset is arranged into 4 abnormal classes namely fighting, snatching, panic running, and kidnapping. It contains complex backgrounds, real-life objects (cycle, motorbike, four-wheeler), both indoor and outdoor environments as well as illumination change. To tackle ambiguity during the transition from normal to abnormal or vice-versa we conducted voting (subjective evaluation) with 10 volunteers. We further augmented the dataset with two of the most common weather phenomena namely haze and rain to bridge the gap between real-world challenges and dataset.
End-to-end pre-trained large vision language models (VLMs) have made unprecedented progress in image captioning. Nonetheless, they struggle to generate detailed captions, which necessitate the models capturing spatial relations, counting, text rendering, world knowledge, and other presenting or not presenting aspects of the image. To overcome their inadequacies, we present a Question – Answer – Caption methodology, named QAC, that performs questioning and answering on many aspects of the given image, followed by captions based on the responses. Specifically, we use ChatGPT to produce a set of questions about the images’ content. The questions are then answered using a pre-trained VLM. After gathering all answers, we prompt the pre-trained VLM to generate descriptive captions in a zero-shot setting. Our approach is plug-and-play and can be easily applied on any pre-trained VLM. We implement QAC on InstructBLIP and LLaVA, demonstrating comparable performance to fine-tuned models on a challenging DOCCI dataset.
