In this study, we investigate the Deep Image Prior (DIP) in enhancing image smoothing, a crucial component in numerous computer vision and graphics applications. Although deep learning has demonstrated remarkable achievements in these domains, it often falls short in flexibility and controllability, in contrast to traditional methods, which are more adaptable and typically exhibit subpar performance. Notably, some end-to-end deep learning models offer control over edge preservation, yet their performance remains marginally suboptimal. To address this shortcoming, we introduce an innovative network architecture that diverges from the traditional U-Net model, featuring a Laplacian pyramid as the encoder and a deep decoder as the decoding component, integrated with a bilateral filter loss to improve DIP. This design aids the network in rapidly assimilating essential low-frequency information. Our approach excels in retaining texture details, significantly improving image smoothing and related tasks beyond the capabilities of standard DIP methods. Moreover, our technique outperforms the leading unsupervised method, pyramid texture filtering, in texture filtering tasks and other applications.
Computer vision tasks such as object detection and segmentation rely on the availability of extensive, accurately annotated datasets. In this work, We present CIA, a modular pipeline, for (1) generating synthetic images for dataset augmentation using Stable Diffusion, (2) filtering out low quality samples using defined quality metrics, (3) forcing the existence of specific patterns in generated images using accurate prompting and ControlNet. In order to show how CIA can be used to search for an optimal augmentation pipeline of training data, we study human object detection in a data constrained scenario, using YOLOv8n on COCO and Flickr30k datasets. We have recorded significant improvement using CIA-generated images, approaching the performances obtained when doubling the amount of real images in the dataset. Our findings suggest that our modular framework can significantly enhance object detection systems, and make it possible for future research to be done on data-constrained scenarios. The framework is available at: github.com/multitel-ai/CIA.
Large language models (LLMs) have exhibited impressive capabilities in human-like reasoning and generation, prompting widespread exploration of their utility in mathematical problem-solving. However, existing research predominantly focuses on text-based mathematical challenges, neglecting those involving geometric concepts. To bridge this gap, our research aims to empower LLMs to tackle geometric problems by integrating image understanding. We investigate the limitations of current Multimodal Large Language Models (MLLMs) in this context, particularly their struggle to accurately interpret complex geometric elements and their mathematical reasoning. To support this endeavor, we introduce GeoVQA, a specialized multimodal math dataset tailored for secondary school education. Furthermore, we conduct experiments to Instruction-tune LLaVA [14] and G-LLaVA [11] models, evaluating their performance against benchmarks such as PGPS9K [21] & Geometry3k [15]. Additionally, we compare the effectiveness of Gemini pro [19] and GPT4 [1] in handling geometric tasks using various prompting techniques, employing metrics like accuracy and human evaluation for meticulous verification. GPT-4 has the best performance on zero-shot inference on our dataset. LLaVA-v1.5-13B model finetuned on our dataset outperforms GPT-4 on PGPS9K benchmark. GitHub: GeoVQA.
Hand Gesture Recognition (HGR) is an important component of multimedia solutions and human-computer interaction systems. Utilizing multimodal information, e.g. from both the camera and depth sensors, to improve the recognition accuracy, usually comes at the cost of additional power consumption. In this paper, we introduce a novel framework for cross-modality adaptive sensing and perception (CM-ASAP) using multimodal Deep Neural Networks (DNNs). Inspired by human cognition and multimodal perception, our proposed framework dynamically allocates computation and sensing resources between different modalities to optimize the trade-off between efficiency and accuracy of multimodal HGR DNNs. CM-ASAP accounts for the dominance of sensor power consumption over computational costs, as well as for the variance in the effectiveness of modality fusion. Furthermore, our proposed CM-ASAP takes advantage of potential early gesture classification based on initial frames.
Information processing and retrieval in literature are critical for advancing scientific research and knowledge discovery. The inherent multimodality and diverse literature formats, including text, tables, and figures, present significant challenges in literature information retrieval. This paper introduces LitAI, a novel approach that employs readily available generative AI tools to enhance multimodal information retrieval from literature documents. By integrating tools such as optical character recognition (OCR) with generative AI services, LitAI facilitates the retrieval of text, tables, and figures from PDF documents. We have developed specific prompts that leverage in-context learning and prompt engineering within Generative AI to achieve precise information extraction. Our empirical evaluations, conducted on datasets from the ecological and biological sciences, demonstrate the superiority of our approach over several established baselines including Tesseract-OCR and GPT-4. The implementation of LitAI is accessible at https://github.com/ResponsibleAILab/LitAI.
We evaluate the presence and cybersickness scores of Virtual Reality (VR) applications using a Head-Mounted Display (HMD) with different: (i) 3D objects in non-neural and neural representations and (ii) interaction modes in 0-, 3-, and 6-DoF (Degrees-of-Freedom). The presence score indicates how much an HMD user feels like they are there, while the cybersickness score represents how much an HMD user reports discomfort. These two scores impose crucial impacts on VR experience in HMDs. To the best of our knowledge, this paper is the first work investigating the VR Quality-of-Experience (QoE) of the latest neural 3D object representations, including 3D Gaussian Splatting (3DGS). The results from our user study reveal the benefits of 3DGS, it: (i) delivers the highest presence scores; (ii) achieves the lowest cybersickness scores; (iii) reaches the full frame rates; and (iv) consumes the least storage space. We also found that the 6-DoF interaction mode increases the presence scores and reduces the cybersickness scores, compared to 0- and 3-DoF interaction modes in most cases. The findings from our user study shed some light on future investigations into many other factors of VR QoE in HMDs.
In the pursuit of computation efficiency, leveraging multiple compute devices presents a significant opportunity for performance enhancement of locally running Large Language Models (LLMs) on commodity consumer devices, such as laptops. This paper presents a method to improve Time To First Token (TTFT) performance of llama. cpp[1], a popular framework used throughout the industry to execute a wide array of state-of-the-art LLMs locally, by employing a heterogeneous computing approach. We introduce a novel parallelization strategy that harnesses the combined compute of CPU, GPU, and Neural Processing Unit (NPU) to significantly reduce TTFT latency. Our methodology involves a detailed analysis of llama.cpp to identify parallelizable segments, followed by the implementation of a heterogenous execution model using OpenVINO (TM)[2]. The results demonstrate a similar to 6X performance improvement in TTFT latency when using our heterogenous compute approach, as compared to the native implementation of llama.cpp running on CPU. This study not only showcases the potential of multi-device computing in enhancing llama.cpp, but also serves as a blueprint for similar optimizations in other compute-intensive applications.
Video stabilization is essential for producing high-quality videos by eliminating unwanted camera motion, resulting in smooth, steady footage that enhances visual coherence. However, existing methods often pose several challenges such as the difficulties in accurately tracking specific objects, and high computational demands. These limitations hinder the ability to achieve optimal stabilization results and limit the practical applicability of video stabilization techniques in real-world scenarios. To address these challenges, we introduce a novel end-to-end method for object-centric video stabilization in the presence of multiple independent objects. Our approach introduces two algorithms: 1) Object-Centric Video Stabilization (OCViS), which facilitates effective detection and tracking of specific objects within each frame, followed by stabilization with respect to the object of interest, and 2) Optical Flow-based Transformation Accumulation and Smoothing (OFTrAnS) to remove inherent shakiness and maintain consistent stabilization, even with multiple independent objects. Most importantly, our algorithms achieve these results without relying on deep neural networks or ground-truth data, providing a more efficient and accessible solution for enhancing video quality on mobile devices and handheld cameras. Our experiments demonstrate that the proposed approach significantly outperforms a strong baseline by a large margin emphasizing the efficacy of our method.
A lightweight, high-performance, and unsupervised method to reconstruct the surface from 3D point cloud data, named green point cloud surface reconstruction (GPSR), is proposed in this work. GPSR's core concept involves building a rough signed distance field (SDF) and iteratively refining it. The method comprises three modules: 1) rough SDF initialization, 2) iterative SDF refinement, and 3) surface extraction from refined SDF. Initially, GPSR employs a voxel representation and utilizes a heat diffusion process, implemented through Gaussian kernel convolution, to establish an initial SDF. In the refinement stage, additional heat sources and sinks are introduced around points to further refine the SDF. Finally, the surface is extracted from the temperature grid. The paper's contribution lies in leveraging the approximated heat diffusion to produce surfaces under low complexity. Extensive experiments demonstrate GPSR's robust performance compared to other benchmarking methods across various types of point set degradation.
With the rise of digital content on social media and the advancement of image editing tools, tampering with scene text has become a serious concern. Scene text manipulation detection (STMD) is a kind of image manipulation detection (IMD) with focus on the tampering of scene text pixels, which is crucial for image content integrity and media forensics. In this paper, we present TextSleuth, a novel benchmark dataset specifically designed for STMD, by integrating three public datasets with newly introduced manipulation and annotations. We introduce professional edits on the Total-Text dataset (similar to 1K images) with four levels of manipulated region perceptibility, and a large synthetic manipulation set (858K images) on the SynthText dataset, as well the integration of the Tampered-IC13 dataset (378 images). We established a new STMD baseline based on TextSleuth using MMFusion-IML, the state-of-the-art image manipulation detection model. We performed extensive experiments, reporting the AUC from ROC analysis and the balanced accuracy (bACC) metrics to maintain a balanced performance evaluation. The MMFusion-IML baseline achieves 0.641 AUC and 0.588 bACC on the Total-Text subset. In comparison, it achieves 0.89 AUC and 0.8272 bACC on the Tampered-IC13 subset. This showcases the real-world STMD challenges reflected in our new dataset. TextSleuth is a valuable resource for future research in scene text manipulation detection and forensics. The dataset is available at https://github.com/abhineet-pandey/Text- Sleuth.
Effective 3D patient data visualization for brain MRI is required in surgical planning and in-situ guidance during brain tumor resection. Traditional 2D displays do not possess depth perception and are poor in understanding complex anatomical structures being displayed in 3D. In this work, an interactive visualization system based on augmented reality is presented, which provides an intuitive and truly immersive way to visualize and interact with 3D brain MRI data as preoperative planning and in-situ guidance by a surgeon during tumor resection procedures of the brain. In this work, we propose a system where AR Devices can render patient-specific 3D brain models, reconstructed from MRI data, in a spatially registered way to the surgical field. These 3D virtual models can be interacted by the surgeon, through the natural use of controls, to study the spatial relationship of critical structures such as tumors, ventricles, and functional brain regions. Advanced volume rendering and multi-modal data fusions are used in creating photorealistic 3D visualizations with high anatomical detail.
Deep learning methods can now generate high quality synthetic speech which is perceptually indistinguishable from real speech. As synthetic speech can be used for nefarious purposes, speech forensics methods to detect fully synthetic speech have been developed. Speech editing tools can also create partially synthetic speech in which only a part of the speech signal is synthetic. Detecting these short synthetic segments within a speech signal requires specialized methods to determine the tem-poral location of the synthetic speech. In this paper, we propose the Synthetic Speech Localization Convolutional Transformer (SSLCT), a neural network and transformer method for synthetic speech localization. SSLCT can temporally localize synthetic speech segments as small as 20 milliseconds. We demonstrate that SSLCT achieves less than 10% Equal Error Rate (EER), which is an improvement over several existing methods.
In this position paper, we present a vision for the future of filmmaking, driven by the emergence of generative AI technology. While the AI workflow for filmmaking remains in its infancy, recent advancements in diffusion models and the emergence of AI tools such as Midjourney, Runway, Pika, SORA, and LUMA are profoundly inspiring. The integration of AI into filmmaking endeavors, exemplified by projects like Our T2 Remake and Next Stop Paris, has yielded unprecedented impacts. This paper meticulously examines the challenges currently confronting AI models and proposes temporary solutions to surmount these obstacles in the filmmaking process. Furthermore, it demonstrates the workflow of the film “Next Stop Paris,” illustrating how these integrated AI modules can collaborate efficiently to produce short films despite technical limitations in the early days. We foresee a future akin to Silicon Valley's technology incubation, where intellectual property (IP) incubation thrives in Hollywood. This initiative supports our vision of catalyzing 10x growth in the filmmaking industry.
A sequence variant (SV) containing branches is considered an extension of a sequence widely used to represent an ordered list of items. Although comparing such SVs is vital in practical applications, an efficient method to compare more than three SVs efficiently has yet to be studied. When the number of SVs increases, there is a high possibility that common parts do not exist. Hence, we cannot thoughtfully understand the similarities and differences among the SVs that were compared. In this paper, we first develop a method to exclude general items that have high frequency because they appear in almost every sequence and then cluster the SVs into several groups using a defined SV similarity. Finally, we calculate the longest common SV in each group and generate a merged SV to visualize the commonality and differences of the target SVs efficiently. The proposed method is shown to be effective when applied to a real medical dataset from 23 medical institutions.
Automated trading strategies can provide an edge over human trading but are challenging to develop and monitor. In this study, two strategies are investigated and developed, namely a rule-based momentum strategy and reinforcement learning driven strategy. The rule-based momentum strategy uses multiple technical indicators to define profitable entry and exit criteria over time, whereas the reinforcement learning driven strategy combines market and sentiment data to train a machine learning model on how to profit from stock trading. A scalable framework is then constructed to enable the automation of these algorithmic trading strategies by streaming real time data and integrating with a stockbroker API to manage order and accounts.
Volumetric point cloud media streaming has become increasingly popular in recent years. Streaming point clouds over the Internet requires efficient compression and transmission tech- niques to ensure that the client can receive the data in real-time. However, this poses several challenges in terms of bandwidth, latency, and quality of experience (QoE). The current network conditions are characterized by long delays and packet loss due to congestion, which results in the loss of many point clouds during transmission. As a result, clients may see incomplete or distorted objects, leading to low QoE. Researchers have proposed 3D point cloud completion models. These models aim to recover the missing points in an incomplete point cloud. However, all the proposed models in the literature consider incomplete objects from well-known synthetic datasets, which do not mimic packet loss. Therefore, these models fail to recover the missing points when deployed in real point cloud streaming scenarios. To fill this gap, we propose Gilbert-Elliott Shape Attack algorithm to generate a 3D point cloud incomplete dataset, namely GESA, that mimics the packet loss impact. This dataset is designed to be representative of real-world point cloud streaming scenarios, and it allows for the evaluation of the performance of existing completion models under realistic conditions. Our dataset is publicly available at https://in2gm.github.io/GESA/.
Detecting objects in stationary scenes using fisheye cameras poses challenges due to fluctuations in object sizes and distortions at different image locations, which can degrade the accuracy of existing classifiers. To address the unique challenges posed by fisheye cameras in stationary object detection scenarios, we introduce FisheyeAdapt, a novel framework that seamlessly integrates tailored post-processing techniques with distortionaware training strategies, enabling robust and precise object recognition in highly distorted fisheye imagery. We present OmniDet (Scene Context-Aware LEarning for Fisheye), a novel approach that dynamically adjusts confidence thresholds based on object categories and sizes, while leveraging scene context-aware model training. Through extensive experiments, we demonstrate that OmniDet consistently improves the performance of object detection across various fisheye camera-based models, showcasing its wide applicability and effectiveness. Extensive experiments demonstrate the effectiveness of our method.
As generative artificial intelligence (GenAI) systems increasingly influence the fields of art and design, they raise critical challenges regarding copyright and artistic attribution. This paper introduces a novel system for generating visual citations within AI-generated images, thereby addressing copyright concerns while fostering ethical use and sharing of digital art. The proposed system aims to efficiently identify elements and styles that are possibly derived from existing artworks within the outputs of text-to-image AI systems. This system enhances privacy and security as it operates on image features and embeddings, rather than accessing the images directly, thus avoiding the handling of copyrighted materials. The system automatically embeds metadata into the images, detailing the origins of the included artistic elements. Moreover, the system enhances compliance by enabling automatic or manual modifications of generation prompts, ensuring that generated images are free of copyright infringements. This functionality not only protects artists' intellectual property but also supports transparent acknowledgment in digital media creation. The proposed system has the potential to facilitate respectful interactions between AI technologies and creative content.
Multimedia applications for machine learning mod-els are characterized by the fusion of multiple modalities of data. In this work, we highlight the trust and robustness challenges of machine learning that arises from data fusion. To do so, we present three case studies demonstrating how multimedia applications exacerbate existing challenges of trustworthy and robust machine learning. For the first case study, we investigate the impact of fusion depth on the robustness of multi-modal machine learning models, observing that model architecture could impact robustness. For the second case study, we investigate the impact of fusion modality on the robustness of multi-modal machine learning models, observing that fusion models are only as robust as their most susceptible modality. For the third case study, we explore the impact of weight quantization techniques on the robustness of multi-modal models, observing the need for modality-based quantization schemes. Through these case studies, we hope to shed light on the unique trust and security challenges that arise in machine learning models when applied in multimedia applications and offer insights to fortify such systems in real-world scenarios.
Virtual Reality (VR) applications take users' Head-Mounted Display (HMD) and controller trajectories as inputs for an immersive experience. Leakage of these trajectories threatens user privacy in several aspects, including but not limited to their identities. Existing privacy-preserving approaches, however, overlook the temporal correlation of VR user trajectories, which could be leveraged by attackers. In this paper, we develop a disturber to perturb VR user trajectories in both temporal and spatial domains on the fly. Such trajectory perturbations could, unfortunately, lead to distorted rendered VR viewports. Thus, we develop a compensator to recover from such distortion using efficient image-warping algorithms. Our evaluation results show the merits of our proposed solution: (i) our disturber alone reduces at most 0.42 re-identification rate of VR users compared to the state-of-the-art approach, (ii) our disturber alone outperforms the state-of-the-art approach by 2.43 dB in PSNR, 0.13 in SSIM, and 8.15 in VMAF under the same privacy-preserving settings, and (iii) our compensator further boosts the visual quality of a VR application by at most 6.83 dB in PSNR, 0.45 in SSIM, and 34.57 in VMAF, compared to disturber-only solution.
