In this study, we investigate the Deep Image Prior (DIP) in enhancing image smoothing, a crucial component in numerous computer vision and graphics applications. Although deep learning has demonstrated remarkable achievements in these domains, it often falls short in flexibility and controllability, in contrast to traditional methods, which are more adaptable and typically exhibit subpar performance. Notably, some end-to-end deep learning models offer control over edge preservation, yet their performance remains marginally suboptimal. To address this shortcoming, we introduce an innovative network architecture that diverges from the traditional U-Net model, featuring a Laplacian pyramid as the encoder and a deep decoder as the decoding component, integrated with a bilateral filter loss to improve DIP. This design aids the network in rapidly assimilating essential low-frequency information. Our approach excels in retaining texture details, significantly improving image smoothing and related tasks beyond the capabilities of standard DIP methods. Moreover, our technique outperforms the leading unsupervised method, pyramid texture filtering, in texture filtering tasks and other applications.
Computer vision tasks such as object detection and segmentation rely on the availability of extensive, accurately annotated datasets. In this work, We present CIA, a modular pipeline, for (1) generating synthetic images for dataset augmentation using Stable Diffusion, (2) filtering out low quality samples using defined quality metrics, (3) forcing the existence of specific patterns in generated images using accurate prompting and ControlNet. In order to show how CIA can be used to search for an optimal augmentation pipeline of training data, we study human object detection in a data constrained scenario, using YOLOv8n on COCO and Flickr30k datasets. We have recorded significant improvement using CIA-generated images, approaching the performances obtained when doubling the amount of real images in the dataset. Our findings suggest that our modular framework can significantly enhance object detection systems, and make it possible for future research to be done on data-constrained scenarios. The framework is available at: github.com/multitel-ai/CIA.
Large language models (LLMs) have exhibited impressive capabilities in human-like reasoning and generation, prompting widespread exploration of their utility in mathematical problem-solving. However, existing research predominantly focuses on text-based mathematical challenges, neglecting those involving geometric concepts. To bridge this gap, our research aims to empower LLMs to tackle geometric problems by integrating image understanding. We investigate the limitations of current Multimodal Large Language Models (MLLMs) in this context, particularly their struggle to accurately interpret complex geometric elements and their mathematical reasoning. To support this endeavor, we introduce GeoVQA, a specialized multimodal math dataset tailored for secondary school education. Furthermore, we conduct experiments to Instruction-tune LLaVA [14] and G-LLaVA [11] models, evaluating their performance against benchmarks such as PGPS9K [21] & Geometry3k [15]. Additionally, we compare the effectiveness of Gemini pro [19] and GPT4 [1] in handling geometric tasks using various prompting techniques, employing metrics like accuracy and human evaluation for meticulous verification. GPT-4 has the best performance on zero-shot inference on our dataset. LLaVA-v1.5-13B model finetuned on our dataset outperforms GPT-4 on PGPS9K benchmark. GitHub: GeoVQA.
Hand Gesture Recognition (HGR) is an important component of multimedia solutions and human-computer interaction systems. Utilizing multimodal information, e.g. from both the camera and depth sensors, to improve the recognition accuracy, usually comes at the cost of additional power consumption. In this paper, we introduce a novel framework for cross-modality adaptive sensing and perception (CM-ASAP) using multimodal Deep Neural Networks (DNNs). Inspired by human cognition and multimodal perception, our proposed framework dynamically allocates computation and sensing resources between different modalities to optimize the trade-off between efficiency and accuracy of multimodal HGR DNNs. CM-ASAP accounts for the dominance of sensor power consumption over computational costs, as well as for the variance in the effectiveness of modality fusion. Furthermore, our proposed CM-ASAP takes advantage of potential early gesture classification based on initial frames.
Information processing and retrieval in literature are critical for advancing scientific research and knowledge discovery. The inherent multimodality and diverse literature formats, including text, tables, and figures, present significant challenges in literature information retrieval. This paper introduces LitAI, a novel approach that employs readily available generative AI tools to enhance multimodal information retrieval from literature documents. By integrating tools such as optical character recognition (OCR) with generative AI services, LitAI facilitates the retrieval of text, tables, and figures from PDF documents. We have developed specific prompts that leverage in-context learning and prompt engineering within Generative AI to achieve precise information extraction. Our empirical evaluations, conducted on datasets from the ecological and biological sciences, demonstrate the superiority of our approach over several established baselines including Tesseract-OCR and GPT-4. The implementation of LitAI is accessible at https://github.com/ResponsibleAILab/LitAI.
