Denoising-based diffusion models have attained impressive image synthesis; however, their applications on videos can lead to unaffordable computational costs due to the per-frame denoising operations. In pursuit of efficient video generation, we present a Diffusion Reuse MOtion (Dr. Mo) network to accelerate the video-based denoising process. Our crucial observation is that the latent representations in early denoising steps between adjacent video frames exhibit high consistencies with motion clues. Inspired by the discovery, we propose to accelerate the video denoising process by incorporating lightweight, learnable motion features. Specifically, Dr. Mo will only compute all denoising steps for base frames. For a non-based frame, Dr. Mo will propagate the pre-computed based latents of a particular step with inter-frame motions to obtain a fast estimation of its coarse-grained latent representation, from which the denoising will continue to obtain more sensitive and fine-grained representations. On top of this, Dr. Mo employs a meta-network named Denoising Step Selector (DSS) to dynamically determine the step to perform motion-based propagations for each frame, ensuring the correct transformation of multi-granularity visual features. Extensive evaluations on video generation and editing tasks indicate that Dr. Mo delivers widely applicable acceleration for diffusion-based video generations while effectively retaining the visual quality and style. Video generation and visualization results can be found at https://drmo-denoising-reuse.github.io.

Positive-unlabeled (PU) learning aims to train a classifier using the data containing only labeled-positive instances and unlabeled instances. However, existing PU learning methods are generally hard to achieve satisfactory performance on trifurcate data, where the positive instances distribute on both sides of the negative instances. To address this issue, firstly we propose a PU classifier with asymmetric loss (PUAL), by introducing a structure of asymmetric loss on positive instances into the objective function of the global and local learning classifier. Then we develop a kernel-based algorithm to enable PUAL to obtain non-linear decision boundary. We show that, through experiments on both simulated and real-world datasets, PUAL can achieve satisfactory classification on trifurcate data.

Understanding the dynamics of climate variables is critical for sectors like energy and environmental monitoring. This study addresses the pressing need for accurate mapping of environmental variables in national or regional monitoring networks, a challenge exacerbated by skewed data and large gaps. While this may not be immediately apparent, managing skewness across multiple data sources introduces additional complexities, as conventional transformation methods often fail to effectively normalize the data or preserve inter-dataset relationships. Furthermore, the literature highlights that interpolation uncertainty is closely linked to the interpolation distance, making the handling of large gaps particularly problematic. To tackle these challenges, we propose a novel data fusion approach: the warped multifidelity Gaussian process. This method predicts time-series data from multiple sources with varying reliability and resolution, while effectively addressing skewness and demonstrating partial independence from interpolation distance. Through extensive simulation experiments, we explore both the strengths and limitations of the method. Additionally, as a case study, we apply warped multifidelity Gaussian process (WMFGP) to wind speed data from the Agenzia regionale per la protezione ambientale (ARPA) Lombardia network, a regional environmental agency in Italy. Our results demonstrate the efficacy of WMFGP in filling large gaps in wind speed data, providing more accurate predictions that are essential for air quality forecasting, network maintenance.

Automatic adversarial prompt generation provides remarkable success in jailbreaking safely-aligned large language models (LLMs). Existing gradient-based attacks, while demonstrating outstanding performance in jailbreaking white-box LLMs, often generate garbled adversarial prompts with chaotic appearance. These adversarial prompts are difficult to transfer to other LLMs, hindering their performance in attacking unknown victim models. In this paper, for the first time, we delve into the semantic meaning embedded in garbled adversarial prompts and propose a novel method that "translates" them into coherent and human-readable natural language adversarial prompts. In this way, we can effectively uncover the semantic information that triggers vulnerabilities of the model and unambiguously transfer it to the victim model, without overlooking the adversarial information hidden in the garbled text, to enhance jailbreak attacks. It also offers a new approach to discovering effective designs for jailbreak prompts, advancing the understanding of jailbreak attacks. Experimental results demonstrate that our method significantly improves the success rate of jailbreak attacks against various safety-aligned LLMs and outperforms state-of-the-arts by large margins. With at most 10 queries, our method achieves an average attack success rate of 81.8 attacking 7 commercial closed-source LLMs, including GPT and Claude-3 series, on HarmBench. Our method also achieves over 90 Llama-2-Chat models on AdvBench, despite their outstanding resistance to jailbreak attacks. Code at: https://github.com/qizhangli/Adversarial-Prompt-Translator.

Metric-based methods are one of the most common methods to solve the problem of few-shot image classification. However, traditional metric-based few-shot methods suffer from overfitting and local feature misalignment. The recently proposed feature reconstruction-based approach, which reconstructs query image features from the support set features of a given class and compares the distance between the original query features and the reconstructed query features as the classification criterion, effectively solves the feature misalignment problem. However, the issue of overfitting still has not been considered. To this end, we propose a self-reconstruction metric module for diversifying query features and a restrained cross-entropy loss for avoiding over-confident predictions. By introducing them, the proposed self-reconstruction network can effectively alleviate overfitting. Extensive experiments on five benchmark fine-grained datasets demonstrate that our proposed method achieves state-of-the-art performance on both 5-way 1-shot and 5-way 5-shot classification tasks. Code is available at https://github.com/liz-lut/SRM-main.

Metric learning aims to learn a distance metric such that semantically similar instances are pulled together while dissimilar instances are pushed away. Many existing methods consider maximizing or at least constraining a distance margin in the feature space that separates similar and dissimilar pairs of instances to guarantee their generalization ability. In this article, we advocate imposing an adversarial margin in the input space so as to improve the generalization and robustness of metric learning algorithms. We first show that the adversarial margin, defined as the distance between training instances and their closest adversarial examples in the input space, takes account of both the distance margin in the feature space and the correlation between the metric and triplet constraints. Next, to enhance robustness to instance perturbation, we propose to enlarge the adversarial margin through minimizing a derived novel loss function termed the perturbation loss. The proposed loss can be viewed as a data-dependent regularizer and easily plugged into any existing metric learning methods. Finally, we show that the enlarged margin is beneficial to the generalization ability by using the theoretical technique of algorithmic robustness. Experimental results on 16 datasets demonstrate the superiority of the proposed method over existing state-of-the-art methods in both discrimination accuracy and robustness against possible noise.

The population of the world is continuously growing older, leading to more people with dementia who need support while living in their own home. Our RadioMe system was designed to adapt a live radio stream with reminders and music intervention for agitation mitigation for people with dementia living in their own home. In this demonstration we present our prototype, with features to record reminders and schedule them to be played during the live radio stream and a music intervention system when agitation is detected.

Language models (LMs) only pretrained on a general and massive corpus usually cannot attain satisfying performance on domain-specific downstream tasks, and hence, applying domain-specific pretraining to LMs is a common and indispensable practice. However, domain-specific pretraining can be costly and time-consuming, hindering LMs' deployment in real-world applications. In this work, we consider the incapability to memorize domain-specific knowledge embedded in the general corpus with rare occurrences and long-tail distributions as the leading cause for pretrained LMs' inferior downstream performance. Analysis of Neural Tangent Kernels (NTKs) reveals that those long-tail data are commonly overlooked in the model's gradient updates and, consequently, are not effectively memorized, leading to poor domain-specific downstream performance. Based on the intuition that data with similar semantic meaning are closer in the embedding space, we devise a Cluster-guided Sparse Expert (CSE) layer to actively learn long-tail domain knowledge typically neglected in previous pretrained LMs. During pretraining, a CSE layer efficiently clusters domain knowledge together and assigns long-tail knowledge to designate extra experts. CSE is also a lightweight structure that only needs to be incorporated in several deep layers. With our training strategy, we found that during pretraining, data of long-tail knowledge gradually formulate isolated, outlier clusters in an LM's representation spaces, especially in deeper layers. Our experimental results show that only pretraining CSE-based LMs is enough to achieve superior performance than regularly pretrained-finetuned LMs on various downstream tasks, implying the prospects of domain-specific-pretraining-free language models.

People with dementia and their carers are experiencing a complicated and highly personal health journey. The RadioMe system, an adaptive live radio system enriched with reminder possibilities and agitation detection and intervention with personalised calming music, is being developed to support people with mild dementia in their own home. RadioMe is an ongoing, interdisciplinary project, combining expertise on dementia, music therapy, music computation and human computer interaction.

Despite their prevalence in deep-learning communities, over-parameterized models convey high demands of computational costs for proper training. This work studies the fine-grained, modular-level learning dynamics of over-parameterized models to attain a more efficient and fruitful training strategy. Empirical evidence reveals that when scaling down into network modules, such as heads in self-attention models, we can observe varying learning patterns implicitly associated with each module's trainability. To describe such modular-level learning capabilities, we introduce a novel concept dubbed modular neural tangent kernel (mNTK), and we demonstrate that the quality of a module's learning is tightly associated with its mNTK's principal eigenvalue $\lambda_{\max}$. A large $\lambda_{\max}$ indicates that the module learns features with better convergence, while those miniature ones may impact generalization negatively. Inspired by the discovery, we propose a novel training strategy termed Modular Adaptive Training (MAT) to update those modules with their $\lambda_{\max}$ exceeding a dynamic threshold selectively, concentrating the model on learning common features and ignoring those inconsistent ones. Unlike most existing training schemes with a complete BP cycle across all network modules, MAT can significantly save computations by its partially-updating strategy and can further improve performance. Experiments show that MAT nearly halves the computational cost of model training and outperforms the accuracy of baselines.

Few-shot image classification is a challenging problem that aims to achieve the human level of recognition based only on a small number of training images. One main solution to few-shot image classification is deep metric learning. These methods, by classifying unseen samples according to their distances to few seen samples in an embedding space learned by powerful deep neural networks, can avoid overfitting to few training images in few-shot image classification and have achieved the state-of-the-art performance. In this paper, we provide an up-to-date review of deep metric learning methods for few-shot image classification from 2018 to 2022 and categorize them into three groups according to three stages of metric learning, namely learning feature embeddings, learning class representations, and learning distance measures. With this taxonomy, we identify the novelties of different methods and problems they face. We conclude this review with a discussion on current challenges and future trends in few-shot image classification.

This work pursues the optimization of over-parameterized deep models for superior training efficiency and test performance. We first theoretically emphasize the importance of two properties of over-parameterized models, i.e., the convergence gap and the generalization gap. Subsequent analyses unveil that these two gaps can be upper-bounded by the ratio of the Lipschitz constant and the Polyak-{\L}ojasiewicz (PL) constant, a crucial term abbreviated as the \emph{condition number}. Such discoveries have led to a structured pruning method with a novel pruning criterion. That is, we devise a gating network that dynamically detects and masks out those poorly-behaved nodes of a deep model during the training session. To this end, this gating network is learned via minimizing the \emph{condition number} of the target model, and this process can be implemented as an extra regularization loss term. Experimental studies demonstrate that the proposed method outperforms the baselines in terms of both training efficiency and test performance, exhibiting the potential of generalizing to a variety of deep network architectures and tasks.

Stress is one of the factors considerably contributing to older adult’s decreasing overall health. Detecting stress in real-time could aid family members to intervene more timely and keep older adults healthier. However, many stress detection systems are not detecting in real-time, depend on multiple devices, capture a plethora of inconveniently sampled data, or use data from younger adults. In this paper, we built a real-time stress detection system for older adults using only heart beats per minute (BPM), which can be easily obtained with most single, comfortable devices. We collected data from people over 60 (N=15), evaluating the Mannheim Multicomponent Stress Test (MMST) for older adults, then built a machine learning model with a classification performance of 76% (AUC) on BPM alone and tested it in real-time in another experiment, comparing the model’s effectiveness with four different heart rate devices. Detection performance decreased considerably (51%) when using the model in another experiment and could not be used successfully with other devices, while a reduced MMST induced stress comparable to the full test suite.

AbstractThe UK Biobank’s brain imaging data is an essential resource for clinical research, but its cost and difficulty in obtaining limit the imaging study to only 100,000 participants, leaving the majority of UKB subjects without imaging data. However, because imaging-derived phenotypes (IDPs) are heritable, and most UKB subjects have genetic information available, it’s possible to predict IDPs for UKB subjects outside the imaging study using genetic data. To this end, this study systematically developed and evaluated biobank-scale genetic polygenic risk scores (PRS) for 4,206 IDPs from multiple brain imaging modalities and processing pipelines. The results indicate that the majority of IDPs (64.76%, 2,774/4,206) were significantly predicted by PRS developed by subjects with both genetic and imaging data. Moreover, genetically predicted IDPs showed associations with a wide range of complex traits and diseases, with the patterns being consistent across different imaging pipelines. These findings suggest that genetic prediction through PRS is a cost-effective and practical way to make the UKB imaging study more beneficial to a broader population. The PRS data resources developed in this study have been made publicly available through Zenodo and will be returned to the UK Biobank.

Deep neural networks have outperformed humans on some image recognition and classification tasks. However, with the emergence of various novel classes, it remains a challenge to continuously expand the learning capability of such networks from a limited number of labeled samples. Metric-based approaches have been playing a key role in few-shot image classification, but most of them measure the distance between samples in the metric space using only a single metric function. In this paper, we propose a Dual Prototypical Network (DPN) to improve the test-time robustness of the classical prototypical network. The proposed method not only focuses on the distance of the original features, but also adds perturbation noise to the image and calculates the distance of noisy features. By enforcing the model to predict well under both metrics, more representative and robust class prototypes are learned and thus lead to better generalization performance. We validate our method on three fine-grained datasets in both clean and noisy settings.

Few-shot classification aims to classify unseen samples by learning from very few labeled samples. Very recently, reconstruction-based methods have been proposed and shown superior performance on few-shot fine-grained image classification, which, on top of the challenge of few labeled samples, faces the difficulty of identifying subtle differences between sub-categories. In essence, these methods reconstruct unseen samples from few seen samples and use the distance between the original unseen samples and their reconstruction as the criterion for classification. However, as pointed out in this paper, a bias is introduced in the overall distribution between the reconstructed features and original features, which consequently affects the distance calculation and subsequent classification. To address this issue, we propose a new concept of Feature Reconstruction Bias (FRB), which can be computed easily in the training stage without introducing any new parameters. Moreover, we propose to use this bias to correct query features in the test stage, which is shown to increase inter-class distances and decrease intra-class distances. Experiments on four fine-grained benchmarks demonstrate the effectiveness of our approach, with state-of-the-art performance achieved in most scenarios.

Hydro-fracture geometry prediction is of great practical importance for optimizing construction parameters and evaluating stimulation effects. Existing physical simulation methods are computationally intensive. Deep learning-based methods offer fast model inference, yet typically require a large amount of field data for accurate model training and lack model interpretability in explaining the complex physical processes. This work presents a physics-informed surrogate modeling method for hydro-fracture geometry prediction. The proposed method encodes the hydro-fracture physical laws, in the form of partial differential equations, as a loss term to govern the training process of the surrogate model, aiming to alleviate the data requirement for model training. Experimental studies demonstrate that the proposed modeling method effectively reduces the training data requirement and improves model accuracy and interpretability.

Metric learning learns a distance metric from data and has significantly improved the classification accuracy of distance-based classifiers such as k-nearest neighbors. However, metric learning has rarely been applied to categorical data, which are prevalent in health and social sciences, but inherently difficult to classify due to high feature ambiguity and small sample size. More specifically, ambiguity arises as the boundaries between ordinal or nominal levels are not always sharply defined. In this paper, we mitigate the impact of feature ambiguity by considering the worst-case perturbation of each instance and propose to learn the Mahalanobis distance through adversarial training. The geometric interpretation shows that our method dynamically divides the instance space into three regions and exploits the information on the “adversarially vulnerable” region. This information, which has not been considered in previous methods, makes our method more suitable than them for small-sized data. Moreover, we establish the generalization bound for a general form of adversarial training. It suggests that the sample complexity rate remains at the same order as that of standard training only if the Mahalanobis distance is regularized with the elementwise 1-norm. Experiments on ordinal and mixed ordinal-and-nominal datasets demonstrate the effectiveness of the proposed method when encountering the problems of high feature ambiguity and small sample size.

Despite achieving state-of-the-art performance, deep learning methods generally require a large amount of labeled data during training and may suffer from overfitting when the sample size is small. To ensure good generalizability of deep networks under small sample sizes, learning discriminative features is crucial. To this end, several loss functions have been proposed to encourage large intra-class compactness and inter-class separability. In this paper, we propose to enhance the discriminative power of features from a new perspective by introducing a novel neural network termed Relation-and-Margin learning Network (ReMarNet). Our method assembles two networks of different backbones so as to learn the features that can perform excellently in both of the aforementioned two classification mechanisms. Specifically, a relation network is used to learn the features that can support classification based on the similarity between a sample and a class prototype; at the meantime, a fully connected network with the cross entropy loss is used for classification via the decision boundary. Experiments on four image datasets demonstrate that our approach is effective in learning discriminative features from a small set of labeled samples and achieves competitive performance against state-of-the-art methods. Code is available at https://github.com/liyunyu08/ReMarNet .

Metric learning aims to learn a distance measure that can benefit distance-based methods such as the nearest neighbor (NN) classifier. While considerable efforts have been made to improve its empirical performance and analyze its generalization ability by focusing on the data structure and model complexity, an unresolved question is how choices of algorithmic parameters, such as the number of training iterations, affect metric learning as it is typically formulated as an optimization problem and nowadays more often as a non-convex problem. In this paper, we theoretically address this question and prove the agnostic Probably Approximately Correct (PAC) learnability for metric learning algorithms with non-convex objective functions optimized via gradient descent (GD); in particular, our theoretical guarantee takes the iteration number into account. We first show that the generalization PAC bound is a sufficient condition for agnostic PAC learnability and this bound can be obtained by ensuring the uniform convergence on a densely concentrated subset of the parameter space. We then show that, for classifiers optimized via GD, their generalizability can be guaranteed if the classifier and loss function are both Lipschitz smooth, and further improved by using fewer iterations. To illustrate and exploit the theoretical findings, we finally propose a novel metric learning method called Smooth Metric and representative Instance LEarning (SMILE), designed to satisfy the Lipschitz smoothness property and learned via GD with an early stopping mechanism for better discriminability and less computational cost of NN.