Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in vision-language understanding tasks. While these models often produce linguistically coherent output, they often suffer from hallucinations, generating descriptions that are factually inconsistent with the visual content, potentially leading to adverse consequences. Therefore, the assessment of hallucinations in MLLM has become increasingly crucial in the model development process. Contemporary methodologies predominantly depend on external LLM evaluators, which are themselves susceptible to hallucinations and may present challenges in terms of domain adaptation. In this study, we propose the hypothesis that hallucination manifests as measurable irregularities within the internal layer dynamics of MLLMs, not merely due to distributional shifts but also in the context of layer-wise analysis of specific assumptions. By incorporating such modifications, HalluShift++ broadens the efficacy of hallucination detection from text-based large language models (LLMs) to encompass multimodal scenarios. Our codebase is available at https://github.com/C0mRD/HalluShift_Plus.
Traditional supervised 3D medical image segmentation models need voxel-level annotations, which require huge human effort, time, and cost. Semi-supervised learning (SSL) addresses this limitation of supervised learning by facilitating learning with a limited annotated and larger amount of unannotated training samples. However, state-of-the-art SSL models still struggle to fully exploit the potential of learning from unannotated samples. To facilitate effective learning from unannotated data, we introduce LLM-SegNet, which exploits a large language model (LLM) to integrate task-specific knowledge into our co-training framework. This knowledge aids the model in comprehensively understanding the features of the region of interest (ROI), ultimately leading to more efficient segmentation. Additionally, to further reduce erroneous segmentation, we propose a Unified Segmentation loss function. This loss function reduces erroneous segmentation by not only prioritizing regions where the model is confident in predicting between foreground or background pixels but also effectively addressing areas where the model lacks high confidence in predictions. Experiments on publicly available Left Atrium, Pancreas-CT, and Brats-19 datasets demonstrate the superior performance of LLM-SegNet compared to the state-of-the-art. Furthermore, we conducted several ablation studies to demonstrate the effectiveness of various modules and loss functions leveraged by LLM-SegNet.
The detection of change in mild cognitive impairment (MCI) is vital between one visit and another visit for the early-stage detection of Alzheimer's disease (AD). It is associated with changes in the structural and functional aspects. Diagnosis of dementia progression is a challenging but essential step for the planning of early treatment. This paper proposes the cellular automata (CA) framework to identify the changes when the disease progresses from one visit to another. An updated rule using the conduction function of anisotropic diffusion has been proposed to assess the earliest signs of dementia manifestation and progression. The progression of dementia disease in multi-visit subjects for healthy and unhealthy patients has been analyzed by examining the entropy from evolved 2D CA images. The subjects below the reference line in healthy cases lead to the prognosis of the disease, whereas subjects above the reference line in unhealthy subjects are in improved condition. The proposed CA architecture shows the transitions and progressive patterns, thereby offering a method for tracking the evolution of dementia over time.
Ensuring optimal crop protection is essential for sustainable agriculture. Effective pest management and detection are crucial for safeguarding paddy yields and minimizing losses. Traditional pest detection methods rely on visual inspection by human experts, which is time-consuming, labor-intensive, and prone to errors. Recent advancements in computer vision and deep learning have enabled automated paddy pest detection systems, offering more efficient and accurate alternatives. However, applying these technologies in real-world agricultural settings, especially in resource-constrained regions, remains challenging. To address this gap, we curated the TPD-20(Tamil Nadu Paddy Pest Dataset-20) dataset, providing researchers with a resource for developing accurate and reliable pest detection models tailored to local agricultural contexts. The TPD-20 dataset comprises 20,000 high-resolution images of paddy plants annotated with bounding boxes indicating the presence of various pests. This dataset represents a diverse range of pest species and paddy plant conditions, capturing the variability of realworld scenarios. We propose a modified Squeeze-and-Excitation (SE) based YOLO model, designed to enhance detection accuracy and robustness by integrating SE modules into the YOLO architecture. These modules selectively emphasize informative features while suppressing less relevant ones, leading to improved feature representation and model performance. Our model achieved a mean average precision (mAP) of 81.2 % and average classification accuracy of 93.8 %, demonstrating the effectiveness of our approach. This study highlights the potential of leveraging computer vision and deep learning for automated pest detection, significantly advancing crop protection efforts. We emphasize the novelty of our approach, which integrates state-of-the-art techniques tailored for paddy pest management.
With the rapid development of modern generative models, the need for an automated synthetic image detection process has never been greater. Recent works in the field of synthetic image detection focus on improving out-of-distribution (OoD) classification performance and robustness to common image pre-processing techniques. However, in this work, we intend to explore the nature of an intricate counter-forensic attack, i.e., the reconstruction of real images with Diffusion Model autoencoders, which could be used to adversely affect the performance of modern synthetic image detection algorithms. We present a variety of experiments to study the nature of this counter-forensic attack and use the inferences from these experiments to develop multiple algorithms to detect such reconstructed images while attempting to detect real and purely synthetic images accurately. To do so, we make use of trained classifiers that can detect real images, autoencoder-reconstructed images, and purely synthetic images. Furthermore, we combine these techniques to build a novel ensemble algorithm that competes with state-of-the-art (SoTA) algorithms in the 'Real vs. Fake' image detection task, while detecting autoencoder reconstructed images accurately, attaining an accuracy of 99.2% in the multiclass setting.
Despite significant advancements in large-scale text-to-image generation and text-conditioned image editing, appearance transfer remains relatively unexplored. Transferring appearance aims to transfer an object's appearance in an appearance image to an object in the structure image so that background details are preserved and accurately reflect the transferred object's characteristics. Appearance transfer has practical applications in areas like virtual try-on and e-commerce product placement. Existing methods often require fine-tuning text-to-image diffusion models or are not applicable to virtual try-on and e-commerce scenarios, which is not ideal. In this paper, we introduce a Mask-Guided attention mechanism that replaces the existing self-attention in U-net architecture of Stable diffusion [29]. This approach can be easily integrated into the Masactrl [6] framework, enabling appearance transfer without model fine-tuning and suitable for a wide range of applications. Our method uses masks of objects in images to guide the appearance transfer process, with these masks obtained from the Segment Anything Model (SAM) [17]. This integration of SAM-generated masks allows for precise object localization and more accurate appearance transfer. We have conducted comprehensive experiments on transferring various clothing items (shirts, jeans, t-shirts) onto people, as well as transferring sofas into living spaces.
Effective detection of gastrointestinal (GI) bleeding in endoscopic images is essential for accurate medical diagnosis and timely intervention. Despite advancements in deep learning, most work is based on predicting the label without considering the certainty in their predictions, leading to potential diagnostic errors. This paper presents an improved GI bleeding detection pipeline that integrates advanced preprocessing techniques, robust uncertainty estimation using deep ensembles, and a comprehensive classification and subsequent detection framework. The preprocessing steps include Gaussian blurring of green and blue channels and applying Contrast-Limited Adaptive Histogram Equalization (CLAHE) on the L channel of the Lab colour space image. After preprocessing, the classification of frames with Swin Transformers and Class Activation Maps (CAMs) for visual explanations are generated using Ablation-CAM. RT-DETR is used to detect the classified bleeding frames precisely, and the doctor classified normal frames with a high uncertainty estimate. The dataset is part of the Auto-WCEBleedGen V2 Challenge of the IEEE ICIP 2024. The experiments are performed on two different test sets, one similar to training, and The proposed preprocessing with Swin Transformers increased classification accuracy by 6.06%. Uncertainty estimation using deep ensembles with predictive entropy metric improved detection average precision at 50, 50:95 IoU by 2.53% and 2.71%, respectively, compared to without uncertainty estimation. These improvements show that the proposed pipeline is more accurate, reliable and interpretable.
In this research we propose a novel Spatial Attention Module (SAM), incorporated on a Convolutional Neural Network (CNN) model, in order to detect Alzheimer's disease from imbalanced Magnetic Resonance Imaging (MRI) datasets. Early detection of Alzheimer's disease has garnered significant interest in the research community since the emergence of Deep Neural Networks, particularly Convolutional Neural Network (CNN) models. The potential impact of such advancements in healthcare is profound, and it has the potential to improve the accuracy of diagnosis. However, the performance of conventional CNN models is hindered due to class imbalance problems, since most of the medical datasets are found imbalanced in nature. The novelties of this work are as follows: (I) A very lightweight CNN model is proposed in which the number of convolutional filters are deployed considerably less and it is trained from scratch. (II) On top of this CNN, we have leveraged one novel attention module, in spatial domain, in which we have incorporated dilated convolutional filters instead of 7x7 filters. This Spatial Attention Module (SAM) has the ability to extract multi-scale features due to incorporating dilation rate 2, thus, it automatically produces a set of distinct features for Alzheimer's disease detection. As a consequence, this Spatial Attention Module (SAM) enables the CNN model to learn more variety of (multi-scale) features in order to generalize well, thus, it automatically mitigates the class imbalance problem to a certain extent. For the validity, we have conducted a 5-fold cross validation experiment by which we have generated 5 distinct MRI datasets from one 'AD-MRI dataset'. In each dataset, a different testing fold is chosen. Experimental results reveal that by the proposed framework we have achieved 97-98% testing accuracy consistently, throughout all the folds, hence, it proved the generalization ability of the proposed framework. Furthermore, our proposed model outperformed the majority of state-of-the-art models, and it surpassed the performance of a recent trends model, the Pooling based Vision Transformer (PiT), by a substantial margin. All the codes of several experiments along with their graph, and confusion matrices are available on a github link: https://anonymous.4open.science/r/Alzheimer-Disease-Detection/README.md
Biometric systems using human physiological or behavioral characteristics are providing very high secure alternatives for authorization. Previous research studies have demonstrated that brain wave signals have great potential in authentication compared to other biometric traits because Electroencephalogram (EEG) is very sensitive in distinguishing the mental attributes between individuals but it has not been explored for cross stimulation evaluation for user verification. In this work, we investigate the feasibility of using resting state, Visually Evoked Potential (VEP) and Steady State Visually Evoked Potential(SSVEP) brain wave signals generated from video and image stimuli for verification purposes. Specifically, the major goal of this paper is to present the influence of evaluating cross stimulation experiment on the performance accuracy of EEG biometric verification methods. Therefore, to present the benchmark, we introduce a newly built Brain Wave EEG database consisting of 6200 samples acquired in three different Data Acquisition Protocols for 50 subjects and we performed the analysis using four EEG biometric verification methods. Experimental results using the LSTM based deep learning methods demonstrates the superior verification accuracy for both within and cross-stimulation evaluations, providing a benchmark for performance analysis.
Single-image 3D reconstruction is a research challenge focused on predicting 3D object shapes from single-view images, requiring all training data for all objects to be available from the start. In dynamic environments, it's impractical to gather data for all objects at once; data becomes available in phases with restrictions on past data access. Therefore, the model must reconstruct new objects while retaining the ability to reconstruct previous objects without accessing prior data. Additionally, existing 3D reconstruction methods in continual learning fail to reproduce previous shapes accurately, as they are not designed to manage changing shape information in dynamic scenes. To this end, we propose a continual learning-based 3D reconstruction method. Our goal is to design a model that can accurately reconstruct previously seen classes even after training on new ones, ensuring faithful reconstruction of both current and previous objects. To achieve this, we propose using variational distribution from the latent space, which represent abstract shapes and effectively retain shape information within a simplified code structure that requires minimal memory. Additionally, saliency maps preserve object attributes, capturing both local minor shape details and the overall shape structure. We employ experience replay to leverage these saliency maps effectively. Both methods ensure that the shape is faithfully reconstructed, preserving all minor details from the previous dataset. This is vital due to resource constraints in storing extensive training data. Thorough experiments show competitive results compared to established methods, both quantitatively and qualitatively.
Air quality estimation through sensor-based methods is widely used. Nevertheless, their frequent failures and maintenance challenges constrain the scalability of air pollution monitoring efforts. Recently, it has been demonstrated that air quality estimation can be done using image-based methods. These methods offer several advantages including ease of use, scalability, and low cost. However, the accuracy of these methods hinges significantly on the diversity and magnitude of the dataset utilized. The advancement of air quality estimation through image analysis has been limited due to the lack of available datasets. Addressing this gap, we present TRAQID - Traffic-Related Air Quality Image Dataset, a novel dataset capturing 26,678 front and rear images of traffic alongside co-located weather parameters, multiple levels of Particulate Matters (PM) and Air Quality Index (AQI) values. Spanning over multiple seasons, with over 70 hours of data collection in the twin cities of Hyderabad and Secunderabad, India, the TRAQID offers diverse day and night imagery amid unstructured traffic conditions, encompassing six AQI categories ranging from “Good” to “Severe”. State-of-the-art air quality estimation techniques, which were trained on a smaller and less-diverse dataset, showed poor results on the dataset presented in this paper. TRAQID models various uncertainty types, including seasonal changes, unstructured traffic patterns, and lighting conditions. The information from the two views (front and rear) of the traffic can be combined to improve the estimation performance in such challenging conditions. As such, the TRAQID serves as a benchmark for image-based air quality estimation tasks and AQI prediction, given its diversity and magnitude. Dataset Link
Segmentation and anatomical landmark detection are important tasks needed for digital planning of surgery. The accuracy of the output from these tasks greatly influence the outcome of downstream tasks such as digital templating and implant size selection. Most of the existing methods treat these tasks separately. However, they are inherently inter-related and can benefit when handled together. In this work, we propose a multi-task network which simultaneously segments and detects the anatomical landmarks in 3D images. The proposed global context aware multi-task network allows for excellent predictions even under limited data. We apply our method on 3D CT volumes of pelvis and through experimental results show that each of the tasks benefit from joint learning, thus providing better performance when compared to dedicated network for each task.
In Visual Question Answering (VQA), the ability to effectively interpret images and questions across diverse domain distributions is imperative. Despite the critical need, there remains a significant gap in methodologies specifically tailored for Cross-Domain VQA (CD-VQA). Current methods predominantly retrofit standard VQA models with domain alignment mechanisms but often fail to address the subtle semantic and visual variances at both micro and macro scales. To overcome these challenges, we introduce a novel strategy, CS-CLIP, which leverages the generalizability of the pre-trained multi-modal foundation model, CLIP, integrated with commonsense knowledge. This integration aims to better bridge the visual and semantic nuances. We enhance the scene graph representations of images by amalgamating CLIP’s visual features with semantic contexts from a pre-trained knowledge graph, thereby producing improved node embeddings. We employ a trainable graph attention module to pinpoint crucial nodes within the scene graph. Additionally, we introduce a cross-attention module to assess the similarity between the question tokens and the scene-graph nodes, generating more directed feature embeddings for the answer decoder. Our comprehensive experiments across multiple benchmarks vividly demonstrate the efficacy of CS-CLIP, significantly enhancing the model’s ability to generalize across domains for VQA.
This paper presents a novel approach to computational art focusing on mandalas—an iconic heritage of Indian art that has proliferated significantly in recent times. Our innovative software allows users to input a handcrafted mandala and select specific motifs for error rectification. The rectification leverages vector information for geometric discretization alongside a simple GCD rule, adhering to traditional mandala principles. The rectified image, being in vector form, allows various real-time operations in the vector space, such as insertion, deletion, or modification of motifs or layers, facilitating enhancements, enlargements, or creation of new compositions. We demonstrate the software’s merit and versatility through various examples, highlighting its potential for special applications such as digital artistry with mandalas, including teaching and training. This work not only advances the field of computational art but also promises to preserve and enhance the rich tradition of mandala art through modern technology.
In the domain of gaze estimation, the quest for a model that is both lightweight and highly accurate presents a considerable challenge. Existing multi-stream architectures, while innovative, often do not fully exploit facial landmarks to accurately represent the intricate geometric relationships between the eyes and head. Furthermore, these models frequently lack an efficient personalization mechanism, which is crucial for achieving optimal accuracy on an individual basis. To address these issues, this paper introduces the Lightweight Adaptive Gaze Estimation (LiAGE) model, offering a breakthrough in tackling these persistent obstacles. Our approach innovatively leverages the detailed facial structure to enhance gaze estimation, ensuring an effective integration of relevant features. Additionally, we introduce a novel lightweight personalization technique designed to significantly improve model precision with a minimal number of calibration steps, requiring only 2 to 10 instances. Our comprehensive experimental validation, conducted using the GazeCapture dataset for 2D gaze estimation, demonstrates the superior performance LiAGE, achieving a 3x reduction in parameters compared to the other state-of-the-art methods.
One of the major challenges in no-reference (NR) image quality assessment (IQA) is the ability to generalize to diverse quality assessment applications. Recently, multi-modal vision-language models are found to be very promising in this direction. They are beginning to form a part of several state of the art NR IQA methods. On the other hand, multi-modal large language models (LLMs) are increasingly being studied for various computer vision applications including IQA. In this work, we perform a thorough study of the ability of multi-modal LLMs for NR IQA by training some of its components and testing for its generalizability. In particular, we keep the LLM frozen and learn parameters corresponding to the querying transformer, LLM prompt and some layers that process the embedding output by the LLM. We observe that some of these components offer a generalization performance far superior to any existing NR IQA algorithm.
Amyloid deposition is considered as a viable biomarker for Alzheimer’s Disease(AD). It is an expensive and less used modality in clinics to study amyloid spatial distribution in brain as the purpose of amyloid imaging is limited to early detection of AD. In this work, we are proposing a novel approach to synthesise amyloid PET images from more commonly used FDG-PET. The proposed method explores the connection between amyloid deposition and associated alteration in glucose metabolism for synthesising amyloid PET using a diffusion model inspired architecture. In addition to the biological connection, proposed network is designed to learn the underlying amyloid deposition patterns with respect to age, gender and status of the disease. The proposed network synthesizes high-quality images and outperforms existing baselines. When tested on classifier trained with real images for Alzheimer’s detection, the synthesized images demonstrate only a marginal decrease in accuracy compared to the classifier tested with real images. This work validates the biological connection between the glucose metabolism and amyloid deposition.
High-intensity activities in sports like basketball can result in fatigue without proper recovery. This study introduces a collaborative framework that leverages Computer Vision (CV) and Machine Learning for evaluating jump landings and predicting athletic readiness by modelling Countermovement Jumps (CMJs) biomechanical aspects. Seventeen female collegiate basketball athletes of Sacred Heart University (SHU), CT, USA, participated in weekly CMJs over a 26-week season. Through CV-driven semantic analysis of videos, the framework identifies the crucial initial contact and maximum flexion point during jump landings and extracts kinetic and kinematic features of the lower extremities. Next, an inferential analysis is conducted to understand the relationship between these features and the CMJ-driven reactive strength index modified (RSImod) score, which measures fatigue and athletic readiness. An XGBoost regressor, trained on the past week’s data, then predicted the RSImod score for the following week, which resulted in an MSE of 0.020 and an R2 of 0.892. Using SHapley Additive exPlanations (SHAP), the framework offers interpretable feedback, aiding coaches in creating personalised training programs and optimising athletic performance while minimising injury risks.
This paper presents a self-supervised learning (SSL) based framework, specifically designed for Handwritten Mathematical Expression Recognition (HMER). The proposed approach incorporates a momentum encoding technique and a non-linear projection head into the image encoder to effectively address the common issue of dimensional collapse in self-supervised learning (SSL) methods. Our approach consists of two main steps: first, we use self-supervised pre-training to train the image encoder to obtain strong feature representations from HME images. Subsequently, we fine-tune the model using a Transformer network to predict LaTeX sequences from HME images. The assessment demonstrates that our SSL framework surpasses other existing SSL frameworks as well as several supervised methods in terms of performance. The findings indicate that our approach is highly advanced, emphasizing its potential to enhance the robustness and efficiency of feature representations in HMER tasks. The integration of momentum encoding and a non-linear projection head in the image encoder is shown to enhance the durability and effectiveness of feature representations, leading to superior performance in HMER tasks. Our experiments reveal that our approaches achieve an expression recognition rate (ExpRate) of 62. 17%, 61. 03%, 64. 8% on the CROHME 2014,2016,2019 test datasets respectively. The CROHME 2019 test data achieves the highest ExpRate, which is state-of-the-art (SOTA). This success is achieved by overcoming the challenges of dimensional collapse and leveraging the advantages of both self-supervised and supervised learning.(1)
Long-Tailed Class Incremental Learning(LTCIL) presents significant challenges due to the inherent class imbalance within tasks and the evolving nature of classes across tasks. Specifically, the under-representation of tail and old classes leads to poor feature representations and classification performance, exacerbating catastrophic forgetting, while overfitting of head and new classes hampers generalization and causes bias. Additionally, adding new classes causes representation drift, misaligning the model's features with older classes. This paper introduces a novel two-stage framework that tackles these challenges by controlling feature space augmentation and rectification using statistical properties. In Stage 1, we propose a Covariance-Controlled Feature Space Augmentation (CCFSA) that generates new feature vectors using the covariance matrix of class distributions, enhancing the representation of tail classes and ensuring better generalization. An Adaptive Class-Specific Margin Loss (ACSM) is also employed to dynamically adjust classification margins, ensuring improved feature separation and reducing drift in feature representations across tasks. Stage 2 rectifies classification boundaries through adaptive global covariance adjustments, fine-tuning the classifier using statistical information from well-represented head classes to balance performance across all classes. Our approach demonstrates superior performance in handling both class imbalance and catastrophic forgetting, as validated through extensive experiments on CIFAR-100 and ImageNet-Subset datasets, outperforming state-of-the-art methods across various long-tail settings.
