DCompared to large language models, traditional multimodal sentiment analysis frameworks are constrained by their classification heads, resulting in poor performance on zeroshot tasks. Moreover, due to limitations in visual encoders and multimodal fusion modules, most existing frameworks can only process a small number of images, leading to a loss of visual information. In light of these issues, this paper proposes a new framework, MSACC. This framework enhances the model's zero-shot performance by adopting a contrastive classification method and reduces the loss of visual information through visual relation extraction and three-dimensional sentiment analysis. We conducted extensive experiments on the Yelp dataset. The experimental results show that MSACC outperforms models of the same category in zero-shot MSA tasks, achieving a 48% performance improvement. Furthermore, compared to the large language model ChatGLM2-6B, MSACC still achieved a 7% performance increase while saving 90% of the model size. In addition, in supervised tasks, MSACC also achieved a 3.27% performance improvement compared to the baseline model.
The task of knowledge graph entity typing (KGET) aims to infer the missing types for entities in knowledge graphs, which is a significant subtask of knowledge graph completion (KGC). In despite of its progress, we observe that the sparsity of the dataset greatly affects the task itself as well as downstream tasks. In this paper, we propose a framework to alleviate this problem, which consists of data augmentation and type inference. We introduce a Statistics-based Entity Type Data Augmentation (SET-DA) method in data augmentation phase, which calculates a type probability distribution for each entity by statistically determining the global relation-type statistics information, and then employ embedding-based KGET models for type inference. Experimental results on two widely used datasets indicate that the proposed framework can solve the data sparsity problem in the task of KGET and the performance of models trained with data augmentation by SET-DA significantly outperforms previous state-of-the-art methods.
Optical image dehazing is a challenging task. Although both physical model-based and deep learning-based dehazing methods have achieved a certain degree of restoration results, they are all reflected in a single spatial domain processing, and deep learning methods have the problem of large models. We start from the flow of information, aiming to combine pixel and frequency domain information, redesign the up- and downsampling mechanism based on discrete wavelet transform, and separate the frequency spectrum of features in the neck network for auxiliary processing, making full use of frequency domain information to assist the dehazing process. Considering the basic nature of the dehazing task in image processing, we introduce feature fusion guided by brightness and texture information to restore higher quality clear and haze-free images. We conducted experiments on SOTS-Indoor, NH-haze, and Dense-haze datasets. The results show that on the Indoor dataset, as shown in Fig.1, Our model parameters and computational costs are reduced by at least 60%, achieving a 41.11dB effect with only 16.48GFlops of computational cost and 1.89M parameters.
As the Internet's complexity and information volume surge, the need for efficient web automation becomes critical. Traditional web agents struggle with redundant web content, which disrupts their understanding of the environment. They also face inefficiencies in multi-task scenarios due to handcrafted exemplars and encounter error accumulation in long-horizon tasks, exacerbated by web-specific complexities like nested structures and interactive elements. To address these issues, we introduce WebSurfer, a novel web agent designed to filter, learn, and adapt in complex environments. WebSurfer refines task-oriented states for clearer observations and employs an exemplar retrieval and ordering strategy to enhance LLMs' understanding and adaptability to current tasks. Notably, WebSurfer features a novel web-wise insight feedback mechanism that enables continuous adaptation and strategy refinement. Evaluations demonstrate that WebSurfer outperforms state-of-the-art (SOTA) methods on realistic tasks, achieving higher accuracy and enhancing longterm adaptability.
Many signal processing and machine learning applications are framed as constrained optimization problems with positive definite constraints. Important examples include kernel matrix learning, covariance estimation of Gaussian distributions, maximum likelihood parameter estimation of elliptically contoured distributions, parameter estimation in Gaussian mixture models and matrix square root. In this work, utilizing Riemannian geometric principles, we propose coordinate-descent algorithms on the Riemannian manifold of symmetric positive definite (SPD) matrices. We further identify a broad class of functions encompassing all the aforementioned applications. Finally, we demonstrate that for this class of functions, the proposed algorithm achieves per-iteration complexities of O(n)-an order of magnitude lower than the O(n(3)) or higher complexities of full update algorithms like Riemannian gradient-descent and interior-point methods. Interestingly, the proposed algorithm is significantly faster than the Burer-Monteiro factorizationbased coordinate descent algorithm, which has a per-iteration complexity of O(n(2)), as confirmed through simulations on the log-det minimization problem.
Speech signals contain rich information, such as textual content, emotion, and speaker identity. To extract these features more efficiently, researchers are investigating joint training across multiple tasks, like Speech Emotion Recognition (SER) and Speaker Verification (SV), aiming to improve performance by decoupling task-specific knowledge. Traditional multitask decoupling methods in SER typically use orthogonalization to increase the distance between parameter vectors in the feature space. In this paper, we introduce a novel Hybrid instance-level Contrastive Decoupling Loss. This method leverages supervised labels to effectively decouple SER and SV. Unlike previous approaches, it is not restricted to dual-stream models with identical architectures and can be easily integrated with leading models for each sub-task. Experimental results show that our proposed Hybrid Contrastive Learning Decoupling (HCLD) method significantly outperforms traditional orthogonal decoupling approaches.
In response to the security risks posed by the realistic propagation of manipulated media data, detecting and grounding multi-modal media manipulation has received attention as a challenging task. However, there is multi-modal contribution imbalance on current approach for cross-modal learning, which affects model performance optimisation. To this end, we propose an Adaptive Contribution Modulation (ACM) framework to solve the problem of multi-modal contribution imbalance. To balance the image and text embedding features before fusion, we propose adaptive weight decision to computes dynamic weights for fusion features, which enable more adaptive and robust decision-making. Meanwhile, we propose contribution modulation block, which dynamically governs the contributions of different modalities for optimization. Based on cross-modal contrastive learning, we balance image and text embeddings contribution through multi-modal contribution balanced learning, which makes better use of the semantic correlation of all modalities. We conduct experiments on the DGM4 dataset, which demonstrate the superior performance of our approach through compared to state-of-the-art methods.
Masked Image Modeling (MIM), following "mask-andreconstruct" scheme, is a promising self-supervised method to learn scalable visual representation. Studies indicate that selecting an effective mask strategy is vital for MIM. However, existing approaches often rely on static pre-defined priors, which limit their ability to adapt mask strategies dynamically for network optimization. In this paper, we focus on the learning process of the network and introduce human-like curriculum into MIM for dynamic representation refinement, and propose an end-to-end framework Curriculum Masked Image Modeling (CurMIM). CurMIM consists of two components: Mask Priority Measurer, which acts as a curriculum learner to determine mask priority values using the network's intrinsic state information, and Dual Adaptive Selector, which serves as a curriculum scheduler to create effective masks based on these values. With negligible extra parameters, our curriculum-based method consistently establishes noticeable improvements across varying model sizes and benchmarks, showing effectiveness and generalization.
Open Information Extraction (OIE) aims at extracting the relational triplets from open-domain texts. Existing methods, unfortunately, mostly fall prey to the complex OIE setting, due to the failure to extract unseen words and underutilize syntactic features. In this work, we propose a novel system tailored for complex OIE, where a generative PLM with non-autoregressive generative decoding is adopted for abstractive OIE generation. We introduce a heterogeneous syntactic forests as features for the task, merging constituency and dependency forests into a unified syntax graph, aiding in better detecting potential boundaries and relationships within complex terms. Also an Implicit-Explicit Contrastive Learning mechanism is devised to boost the perception of implicit relations and spans. Our system outperforms the current state-of-the-art model across seven OIE datasets.
Zero-shot quantization (ZSQ) has emerged as an effective method to reduce model complexity and memory footprint without using original training data, thereby mitigating data privacy and security concerns during model deployment. Recently, Large-Kernel Convolutional Neural Networks (LKC-NNs) have achieved state-of-the-art performance on various vision tasks, which introduce challenges in terms of increased parameters and network complexity, making them difficult to deploy on resource-constrained edge devices. Despite the success of ZSQ, existing methods fail to apply to LKCNNs due to architectural differences such as Batch Normalization (BN) layers in models and thus result in significant performance declines. In this paper, we propose a novel ZSQ framework tailored specifically for LKCNNs, considering their two key characteristics: the large receptive field and the reliance on shape bias. Correspondingly, we first employ an edge detection-based loss to optimize synthetic images that closely mimic the distribution of real images, and a diversity self-distillation loss to maintain consistency in feature representation to enable the generation of synthetic images. Afterward, we use these synthetic images to fine-tune the quantization parameters with a shape-enhance data augmentation strategy. Experiment results demonstrate the superiority of the proposed framework over existing methods, with significant improvements in maintaining accuracy after quantization across various quantization configurations on the ImageNet dataset.
Prosody transfer in speech synthesis aims to produce natural and expressive speech by replicating the prosody of reference speech. Traditional methods, which rely on same-text, samespeaker references during training, often fail to generalize effectively when applied to different-speaker or different-text scenarios during inference, resulting in degraded quality and speaker leakage. To address these limitations, we propose a novel method that leverages content- and speaker-independent references during training, which are prosodically-aligned-meaning that they are closely matched in rhythm, intonation, and stress patterns with the target speech. Specifically, this approach employs non-target references in training to closely mirror test-time conditions. To ensure effective prosody transfer, unit selection is utilized to choose and concatenate segments that closely match the prosody of the target utterance. Additionally, speaker-specific features are carefully normalized during the target cost computation in the unit selection process to enhance the preservation of the target speaker's identity. Evaluation results demonstrate that our method achieves more consistent performance between training and inference, better preserves the target speaker identity, and generates prosody that is comparable to models trained with ground truth references.
False correspondence removal is a persistent challenge in image feature-matching-based applications, especially in complex scenes. Traditional methods often rely on the consistency assumption to model the motion of correct correspondences, which neglects non-consistent correct correspondences, resulting in suboptimal performance. In this paper, we introduce the NCNet, a novel network designed to address this limitation by fitting the motion of both consistent and non-consistent correct correspondences using the learnable frequency response function. Unlike conventional approaches that focus on pixel-based 2D movements, NCNet utilizes the Motion Fitting Residual Module to estimate the high-dimensional motion, capturing the intricate 3D geometrical variations of matching pairs. To further enhance performance, we propose a loss function that balances the performance between the two types of correct correspondences. Extensive experiments demonstrate that NCNet significantly outperforms existing methods in terms of precision for false correspondence removal. The implementation of our approach is publicly available at: https://github.com/Livsdjo/NCNet-Code.
Reward acts as a signal to guide the agent's learning process in Reinforcement Learning (RL), evaluating and assigning rewards to the agent's actions based on their alignment with goals. Designing reward is challenging in multi-agent environment such as StarCraft II benchmark since agents face credit allocation and role adaptation problems. Recent studies have successfully exploited the language understanding and reasoning capabilities of large language models (LLMs) to learn manipulation tasks. Impressed by the remarkable power of LLMs, this paper employs LLMs as role-specific reward designer for playing StarCraft II, making rewards more flexible and task-oriented. Firstly, we develop an interactive text and multi-agent RL environment to study real-time strategy generation in StarCraft II. Secondly, we use LLMs to interpret the game situation and understand agent roles from user instructions. Then, by assigning appropriate subtasks, LLMs quantify the completion of these subtasks to generate role-specific rewards. Further, credit assignment problem is addressed by introducing dynamic reward weights in value decomposition method. In StarCraft II maps, experiments show that role-aligned RL agents trained with our framework achieve superior policy performance, and win rate results demonstrates the effectiveness of our approach in decision-making for micromanagement and long-term planning.
The end-to-end image dehazing network relies on paired training data. However, there is limited training data available for real non-homogeneous dehazing, which limits the performance of dehazing networks on real non-homogeneous hazy images. To overcome this limitation, we propose a method to augment the training datasets for real non-homogeneous image dehazing. Different with existing methods that introduce augment data from other datasets, we estimate the degradations and synthesize additional hazy images by applying it to other scenes in the same dataset. During the estimation, we proposed a new degradation model based on 3D attenuation coefficient for describing non-homogeneous degradation. To solve the 3D attenuation coefficient, we propose a clue kernel to overcome the scene-dependence of the estimated degradation. The experimental results show that our proposed method can effectively augment the training dataset and improve the performance and robustness of the dehazing network. Compared to state-of-the-art methods, our approach outperforms the best performing method by 4.65% SSIM on the Dense-NH-HAZE dataset.
Specific emitter identification refers to identifying a specific emitter by its radio frequency fingerprint extracted from a given signal. Recently, most current methods for specific emitter identification are usually based on neural networks due to their great success. However, with the increasing complexity of radar systems, the neural networks of these methods are too shallow to extract distinctive fingerprint characteristics of modern emitters, especially for those of the same type, resulting in low identification accuracy. To address this challenge, this paper proposes a novel deep convolutional neural network named SepNet. We observed that the feature representation of each radar signal can be decomposed into two independent parts: a radio frequency fingerprint part capturing features of the specific emitter and a variation part capturing features of the signal content. Based on this insight, SepNet is designed to separate fingerprint information from raw signals and uses it to identify specific emitters with high accuracy. SepNet incorporates a separation block and a classification block to perform feature separation and emitter identification tasks, respectively. In addition, a reconstruction block is instrumented into SepNet to guide the separation process. Experimental results demonstrate that SepNet outperforms four end-to-end deep learning models in terms of identification accuracy, and ablation studies confirm the effectiveness of SepNet's architecture.
Pathological voice detection aims to detect voice disorders from speech samples. With the recent development of self-supervised learning (SSL), most studies in the past years of voice disorder employ that technique. Handcrafted features may suggest better prediction since they contain physical information and are more interpretable than SSL. We evaluated different handcrafted acoustic features and SSL approaches for pathological voice detection tasks using sustained vowels. We extracted 88 and 39 dimensional handcrafted features using openSMILE and Praat feature extractors. For SSL, we evaluated wav2vec 2.0, HuBERT, and WavLM, both for feature extractors and finetuning. Results showed that handcrafted features are consistently competitive with SSL features. An ensemble model combining handcrafted and SSL features achieved the best performance with an F1-score of 0.8739 on the test set, outperforming previous studies on the same dataset under the test set. This finding suggests that handcrafted features are still competitive for voice disorder detection tasks, and combining them with SSL features can further improve performance.
Word embeddings and vector representations for linguistic units are important in Natural Language Processing (NLP) and AI systems. Engineering of such models involves significant effort, large amounts of data, and costly computations. Many models and systems have been released to the general public, enabling significant research and development in the classical NLP domain. We demonstrate how word and text embeddings as n-dimensional dense vectors of n real numbers in classical computers can be mapped to highly compressed quantum states or quantum embedding using log(n) qubits. In similarity experiments performed on classical and quantum computers, we show no significant information loss that would affect vector distance scores representing words’ semantic similarity. The results show that existing NLP embedding models in classical computing environments can be used in quantum computing. We discuss the issues and limitations of this approach in the context of current quantum computing environments.
The harmonium, a free-reed keyboard instrument that originated in Europe, is now widely used for vocal accompaniment in Hindustani music. However, it lacks the ability to produce the characteristic pitch ornamentations of Hindustani music, presenting several limitations to its use. In this paper, we present our ideas about developing a keyboard interface and mapping scheme for a digital harmonium that can be used to perform such pitch ornamentations and overcome the limitations of an acoustic harmonium.
