We develop an optimization algorithm suitable for Bayesian learning in complex models. Our approach relies on natural gradient updates within a general black-box framework for efficient training with limited model-specific derivations. It applies within the class of exponential-family variational posterior distributions, for which we extensively discuss the Gaussian case, for which the updates have a rather simple form. Our Quasi Black-box Variational Inference (QBVI) framework is readily applicable to a wide class of Bayesian inference problems and is of simple implementation as the updates of the variational posterior do not involve gradients with respect to the model parameters, nor the prescription of the Fisher information matrix. We develop QBVI under different hypotheses for the posterior covariance matrix, discuss details about its robust and feasible implementation, and provide a number of real-world applications to demonstrate its effectiveness.
Incomplete datasets, in which some objects contain missing entries in certain dimensions, are pervasive in real-world applications. Most existing clustering methods for incomplete data adopt a two-stage pipeline: first imputing missing entries and then performing clustering. However, both imputation and clustering procedures typically involve multiple hyperparameters, which substantially increases the difficulty of producing reliable clustering results in practice. Although decision-graph-based approaches have been shown to alleviate parameter dependence, existing formulations generally assume that all objects share the same set of observed dimensions, and thus cannot be directly applied to incomplete datasets.To address these limitations, we propose a Single-Dimensional Clustering algorithm (SDC), a parameter-free clustering framework tailored for incomplete datasets. By eliminating the imputation stage and extending decision graphs to incomplete data through dimension splitting and a “partition intersection” fusion mechanism, SDC enables effective clustering without requiring any user-specified parameters. Experimental results demonstrate that, across three evaluation metrics, SDC outperforms baseline algorithms by at least 13.7% (NMI), 23.8% (ARI), and 8.1% (Purity). The source code is publicly available at https://github.com/DJLPKU/single_dimension_clustering.
Domain Adaptation (DA) has made substantial progress in mitigating domain shift across diverse domains. However, many unsupervised DA methods rely on explicit cross-domain feature alignment, which can distort target domain features in the absence of ground-truth labels. In this paper, we propose a Target Self-Guided Framework (TSGF), which strengthens the guidance of the target domain for knowledge transfer, achieving adaptation without explicit cross-domain feature alignment, thereby mitigating the feature distortion problem in the target domain. In TSGF, the target domain offers both instance-level and class-level guidance through contrastive learning and pseudo-labeling, respectively, enhancing adaptation through target domain-specific supervision. Furthermore, a patch-mix strategy is introduced to bridge the domain gap, thereby facilitating the capture of shared knowledge across domains. As the emerging Mamba architecture possesses global receptive fields and superior linear complexity compared to Convolutional Neural Network (CNN) and Vision Transformer (ViT) architectures, we explore the application of the Mamba architecture to domain adaptation. Extensive experiments demonstrate that our method achieves superior performance over state-of-the-art approaches, effectively bridging the source and target domains in challenging DA scenarios.
Given the immutable physiological nature of biometric characteristics and their widespread technological adoption, the development of robust template protection mechanisms has become imperative for biometric systems. However, conventional template transformation methodologies frequently encounter critical limitations: performance degradation, cross-domain instability, and residual information leakage - deficiencies that simultaneously compromise system security and recognition efficacy. This paper proposes a paradigm shift through re-engineering of traditional biometric recognition frameworks, specifically leveraging their training-phase models to develop HKNet (Hybrid Key-set Network). Our novel architecture establishes a hybrid key ensemble where each constituent key encapsulates partial reference information through optimized parameter sharing, and performs segmented feature matching via a cross-attention weight matrix. This enables effective decoupling of one-to-one mapping relationships, ensuring that adversaries cannot infer specific user identities through individual key analysis. Furthermore, we propose a key learning and optimization strategy that integrated by tripartite loss function integrates: auxiliary weighted disparity loss for discriminative feature separation, feature discretization loss for security and discrimination, and distance constraint loss for optimal key-space distribution. Through comprehensive experimentation on five public palm vein datasets, the proposed framework demonstrates not only surpasses existing encrypted-domain methods, but also comparable performance to state-of-the-art non-encrypted systems, and formal security guarantees meeting ISO/IEC 24,745 standards for irreversibility, unlinkability, and revocability. We release our code at https://github.com/7033486/HKNet.
Anchor-based multi-view clustering has gained considerable attention owing to its efficiency in dealing with large-scale data by jointly learning compact anchors and structural anchor graph within a unified framework. Most existing methods mainly focus on improving clustering by prior structural constraints on anchor graph and ignore the explicit structural semantic exploration for anchors, which may be insufficient to ensure that anchors are truly representative and limits their expressiveness in discriminative anchor graph learning. To this end, this study proposes Anchor-to-Graph Structural Co-regularization for Multi-view Clustering (AGSCR-MVC), which collaboratively promotes the structural semantic exploration for anchor and anchor graph in a unified framework. In detail, the AGSCR-MVC introduces bipartite graph Laplacian regularizer to improve the clustering structure of anchor graph. And then, it adaptively constructs the anchor adjacency relationships from anchor graph and enforces the explicit structure of latent anchors. The structural regularization from anchors to anchor graph could be negotiated with each other, collaboratively promoting anchor discrimination and clustering quality. Furthermore, the log-sum sparsity is imposed on anchor graph to enhance local structural information. An efficient optimization algorithm is developed for solving the formulated objective problem of proposed model. Extensive experiments persuasively confirm that AGSCR-MVC achieves a compromise superiority in computational complexity and clustering performance. The source code of AGSCR-MVC could be available at https://github.com/XX106222/AGSCR-MVC.
Grouping songs with similar music compositions plays an important role in streaming services as group characteristics can be used to provide personalized song recommendations and thus enhance user engagement. In its essence, a song is characterized by a categorical sequence consisting of notes from different octaves and the sojourn time of every note. Herein, to cluster songs based on their compositions, we propose a cluster-weighted model with marginal distributions as multivariate first-order Markov models and conditional distributions as mixtures of Gammas. Our proposal enjoys great flexibility in capturing potentially multi-modal behavior of sojourn time. Lastly, we apply our model to cluster piano songs and achieve meaningful results.
We present Diff-KATKG, a novel diffusion-based framework for high-fidelity talking head generation, jointly driven by facial keypoints and action units (AUs). To enable fine-grained motion control under sparse driving conditions, we design a cross-attention-based fusion module that fuses keypoint and AU features into a unified embedding, which serves as the conditioning input to the noise prediction network of the diffusion model. This joint representation effectively captures both pose and expression dynamics, enabling expressive and controllable video synthesis. To further enhance temporal coherence, we introduce a cross-frame feature aggregation strategy that leverages spatiotemporal dependencies from previously generated frames to guide the denoising process. This facilitates smoother transitions and more natural motion across frames. Benefiting from the progressive denoising mechanism of the diffusion model, our approach achieves detailed and stable frame reconstruction, significantly improving perceptual realism and temporal consistency.
In cloud-prone regions, transferring the optical source model to a cloudy target domain remains a challenge for supporting continuous monitoring. Conventional methods address it by either reconstructing the optical modality through cloud removal or utilizing the Synthetic Aperture Radar (SAR) modality. Unfortunately, the former methods require extensive cloud-free references, while the latter ignore valuable information that may still exist in partially cloud-contaminated optical images. Therefore, we aim to develop a multi-modality transfer method for the target domain that includes partially cloud-contaminated optical and SAR data without cloud-free references. However, the varying domain gaps and cloud-cover conditions jointly cause the complex modality imbalance problem, which hinders the complementary effect of multi-modality information. The target model not only globally over-relies on the superior modality while neglecting the inferior one, but the role of each modality also locally changes with the amount of remaining optical information. To solve the problems, we introduce a multi-step transfer strategy (MSTS) with the Information Regulation Mechanism (IRM) based on knowledge distillation. The MSTS employs additional auxiliary models to mitigate the domain gaps and quantifies the superior and inferior states of different modalities for each sample. Building on this, the IRM can dynamically adjust the contributions of each modality at the sample level within the target model’s decision-making process. This further improves its applicability, enabling continuous monitoring in cloud-prone regions. Experiments on simulated and real cloud datasets demonstrated that the proposed method outperforms state-of-the-art methods, yielding an average accuracy improvement of 14.03% in land cover scene classification. The codes developed in this study are available at https://github.com/wangyuze-csu/ESCCS.
With the advancement of large language models (LLMs) and multimodal LLMs (MLLMs), table reasoning has achieved significant progress. However, most existing works focus predominantly on textual or rendered tables, which differ substantially from real-world photographed scenarios under suboptimal conditions such as uneven lighting, blur, and tilted perspectives. This discrepancy creates a significant performance gap for current MLLMs, limiting their applicability in real-world scenarios. To address this critical limitation, we introduce the first comprehensive study on multimodal reasoning for photographed tables. We present a new dataset Photographed Chinese Table Reasoning (PCTR-16K), which contains 4989 photographed tables and 16,318 questions across 9 subjects and 3 difficulty levels. This dataset serves as the first benchmark specifically designed for evaluating reasoning capabilities on tables captured under authentic conditions. To enhance MLLMs’ reasoning over photographed tables, we propose Structure-aware Chain-of-Thought (SCoT), a method that unifies table recognition and reasoning into a single end-to-end generative process. To bolster the structural perception required by SCoT, we further incorporate seven auxiliary table structure understanding (TSU) tasks during fine-tuning. These tasks provide fine-grained supervision across multiple dimensions of table layout and semantics. Extensive experiments on various MLLMs demonstrate that our proposed SCoT and multi-view TSU tasks significantly enhance recognition and reasoning capabilities on photographed tables. For example, LLaVA-Llama3.1 achieves an absolute improvement of 19.53% on the PCTR-16K benchmark (from 44.58% to 64.11%), demonstrating its effectiveness for real-world photographed table reasoning. The dataset is publicly available at https://github.com/PremiLab-Math/PCTR-16k.
Large-scale machine learning models, which are revolutionary techniques of artificial intelligence whose main bottlenecks are huge computational expenditure, time and power, are now primarily trained with stochastic optimization algorithms, particularly stochastic gradient descent (SGD) due to their low computational expenses, simplicity, and scalability. In this work, we develop an adaptive stochastic conjugate gradient (SCG) algorithmic framework, automatically determining two key hyper-parameters, the learning rate and the conjugate gradient coefficient, from the perspective of utilizing the stochastic Steffensen method. Although various variants of SCG have been proposed, the acceleration phenomenon of SCG still remains unknown. This work theoretically proves for the first time that for strongly convex problems, the use of conjugate gradient and the Steffensen method makes stochastic optimization algorithms achieve a lower oracle complexity. Additionally, this work shows how to incorporate the Steffensen method into SCG-based algorithms for multivariate problems readily and effectively, avoiding expensive multivariate divided difference. Extensive experiments on different machine learning problems, including logistic regression (LR), support vector machine (SVM), multi-layer perceptrons (MLP), and ResNet-18, support our theoretical findings and confirm that the resulting algorithms compare favorably with several existing state-of-the-art related stochastic first-order and stochastic second-order algorithms. Both theoretical and empirical superiority shows much promise of stochastic Steffensen-based conjugate gradient in machine learning.
The rapid growth of electroencephalogram (EEG)-based applications has led to an exponential rise in EEG data volume, particularly unlabeled data, which challenges supervised learning techniques like EEG classification. Although several unsupervised EEG clustering studies have emerged to provide solutions for EEG labeling, they focus on shallow models and overlook practical scenarios possessing partial prior information. To this end, we propose a semi-supervised deep clustering method for handling partially labeled EEG signals, termed semi-supervised deep EEG clustering with union constraint learning and contrastive learning (EEGcUCC). To the best of our knowledge, this work represents the first attempt to address semi-supervised EEG clustering with deep learning. In EEGcUCC, an unsupervised contrastive learning-based pre-training paradigm, contrastive pre-clustering, is developed for initializing cluster-friendly embeddings and end-to-end output of cluster assignments. To mitigate label scarcity-induced confirmation bias, we introduce a union pairwise constraint loss that integrates prior hard constraints with soft pseudo-constraints derived from high-confidence samples. Furthermore, the proposed union contrastive learning utilizes a semi-supervised contrastive loss to globally integrates prior samples, class prototypes, and pseudo-labeled samples, facilitating the learning of robust embeddings. Comprehensive experiments on 12 EEG datasets demonstrate EEGcUCC yields superior clustering results, compared to advanced unsupervised and semi-supervised clustering methods.
This paper presents ChipDiff, a novel staged diffusion framework specifically designed for Chinese ink-wash style transfer. While conventional generative models often struggle to capture the intricate monochromatic tonal layering and “bone-work” (structural brushstrokes) of traditional art, we propose a bifurcated sampling strategy guided by tailored loss gradients to reconcile structural preservation with stylistic refinement. Inspired by the traditional artistic workflow of “brushing-then-inking”, our approach decouples the denoising trajectory into two functional phases. In the first stage, we leverage a combination of content-structure, HED-edge, and semantic-divergence losses to anchor the global layout and latent skeleton. The second stage further refines the synthesis by injecting fine-grained ink-wash textures and anisotropic bleeding effects through multi-scale stylistic representation losses. Unlike prior single-stage methods, ChipDiff provides a mathematically grounded and artistically intuitive mechanism for controlled pattern generation. Extensive experiments, supported by our curated Traditional Chinese Ink-Wash Painting (TCWP) dataset and a comprehensive perception-aligned evaluation protocol, demonstrate that ChipDiff achieves superior performance in stylistic fidelity and structural integrity over state-of-the-art baselines. Project code is available at https://github.com/hengliusky/ChipDiff/.
Stroke classification is one of the core tasks in online handwritten document recognition. Traditional methods typically rely on domain experts to manually engineer rich glyph and contextual features. While incorporating valuable prior knowledge and interpretability, this approach suffers from high computational costs. This issue is exacerbated in modern online documents, where increasing stroke density and high-resolution trajectories render manual feature extraction inefficient and slow. Consequently, reliance on handcrafted features bottlenecks recognition speed, while high-connectivity GNNs suffer from over-smoothing. To address these challenges, we propose Auto-LR Graph, a graph pooling-based network for stroke classification in online handwritten documents. Auto-GEmbed learns graph-level spatiotemporal context to replace handcrafted features and improve efficiency. Auto-CPool aggregates important nodes to enhance long-range dependency modeling, while Auto-UCPool restores node representations to reduce redundancy. In addition, Auto-kr dynamically adjusts the graph's ability to capture near-and long-range information during pooling and upsampling. We evaluate our method on four datasets, including CASIA-onDo, IAMonDo, FC, and FA. On the CASIA-onDo dataset, Auto-LR Graph achieves an accuracy of 95.15% (+1.59%) and a recognition speed of 28.29 s/doc (-16.93 s/doc). These results demonstrate the clear advantages of the proposed method in both classification accuracy and processing efficiency.
Orthogonal moments (OMs) and their moment invariants are classical and effective feature descriptors. So far, almost all OMs are generated on the basis of existing orthogonal polynomials or their orthogonality properties. Unlike them, we propose a universal method to derive twin orthogonal bases from the parity of polynomial. These bases therefore enable to define a couple of OMs. Focusing on Hermite polynomials, we firstly derive two classes of orthogonal bases which are orthogonal over the interval from zero to infinity. These orthogonal bases are composed of Hermite polynomials of pure-even and pure-odd degrees. Secondly, twin OMs, namely even and odd Hermite-Fourier moments, are proposed. They are defined in the polar coordinates, taking the derived orthogonal bases as the radial parts of basis functions. Thirdly, we formulate a strategy to design combined Contrast, Translation, Rotation and Scaling (CTRS) invariants from the proposed moments. Moreover, this strategy is universal and thus applicable to all OMs defined in the polar coordinates. Finally, extensive numerical experiments such as image reconstruction, invariant image recognition and template matching are conducted. Experimental results not only prove the correctness of proposed theories, but also demonstrate the superior performance of proposed moments and their invariants.
Endoscopic exposure correction faces unique challenges due to non-uniform illumination degradation. Recently, although deep learning-based data-driven methods yield desirable performance, they do not fully consider the physical priors underlying this specific task and generally require a large number of network parameters, thus limiting their application to resource-constrained devices. Against these issues, we propose a lightweight Dual Exposure Correction framework, named DualEC. Specifically, inspired by the image processing pipeline (ISP), we first introduce a physics-driven branch that parameterizes the physical correction steps through specific module designs for explicit correction. To accomplish the flexible learning of pixel-level adjustment parameters in the ISP, we specifically construct a regional adaptive Mamba (RAMamba) to capture important features and spatial attributes. Besides, to increase the flexibility of the calibration, we further introduce a parallel data-driven implicit correction branch based on frequency decoupled Mamba (FDMamba) as a complement to the explicit correction branch. Then, we propose an illumination-guided selective fusion (IGSF) mechanism that utilizes illumination priors to guide the fusion of two-branch images for accurate enhancement. Based on four different tasks, including endoscopic exposure correction, low-light enhancement, downstream segmentation, and real clinical generalization, extensive experiments comprehensively substantiate the superiority of our DualEC. With only 0.13M model parameters and 4.84G FLOPS that are much lower than most existing baselines, our DualEC still achieves quite competitive performance and demonstrates favorable clinical application potential. The code is available at https://github.com/charonf/DualEC.
Super-resolution (SR) is widely exploited in recognition pipelines where models are supposed to be lightweight and, increasingly, trustworthy under input perturbations and security threats. We address this need with two complementary contributions. First, we propose the Aperture Orientation Spectrum Fusion Network (AOSNet). AOSNet views reconstruction as multi-field fusion. Each AOS block combines an Aperture Pyramid Mixer (APM) for multi-scale receptive fields, an Orientation Selective Gate (OSG) for directional structure, and a Spectrum Subband Aggregator (SSA) for frequency refinement. Second, motivated by the growing demand for secure and reliable super-resolution in pattern recognition systems, we build a robustness-enhanced variant, AOSNet-Sec, by attaching a lightweight feature-space branch composed of a SR Stat Detector (SSD), a Security Detect Head (SDH), and a trainable SR Markov Repair (SMR). SSD aggregates per-channel and edge-contrast statistics to form a calibration-only security prior. Meanwhile, the trainable SMR iteratively updates suspicious feature maps guided by normalized statistics. Experiments on standard benchmarks show that AOSNet-Sec substantially reduces PSNR drop and Learned Perceptual Image Patch Similarity (LPIPS) degradation under strong white-box perturbations, which offers a practical trade-off between reconstruction quality and security-oriented robustness.
Visible-Infrared object detection aims to utilize the complementarity of different modalities to improve the accuracy of object classification and localization in complex environments. However, most existing methods prioritize detection performance while neglecting network complexity, limiting their real-world applications. To this end, we propose a lightweight modal-guided cross-attention fusion network (LCAFNet) for visible-infrared object detection, which is composed of a visible-guided cross-attention block (VG-CAB), an infrared-guided cross-attention block (IG-CAB), and a gated fusion block (GFB). The VG-CAB and IG-CAB utilize attention weights from one modality to guide information aggregation from another modality, enabling cross-modal information interaction and feature fusion from two different perspectives. These two blocks generate complementary features containing visible (VIS) and infrared (IR) information, resulting in a comprehensive and robust multimodal feature representation. Benefiting from the enhanced and complementary features generated by the VG-CAB and IG-CAB, the GFB achieves an adaptive and complete fusion between these features through a gating strategy. Further, shallow VIS and IR features extracted from a dual-branch backbone network are applied to mine and integrate complementary features, which have more spatial and edge information, thus improving the localization and classification capabilities of the detection model. Extensive experiments demonstrate that our proposed LCAFNet obtains better detection performance and lower network complexity compared with other excellent models on five commonly used public datasets. Specifically, on the DroneVehicle dataset, our LCAFNet outperforms the state-of-the-art model by 1.6% mAP50, but has only one-eighth the number of network parameters. The source code for our LCAFNet is accessible at https://github.com/WenCongWu/LCAFNet.
Crowd counting and localization under low-visibility conditions (e.g., haze, rain, snow, and nighttime) remain challenging due to degraded image quality, high computational costs, and limited training data. In this paper, we propose a multidimensional knowledge distillation (MDKD) framework designed to achieve a high-precision crowd analysis while maintaining a lightweight architecture. We introduce a plug-in image-restoration-guided (IRG) branch into the teacher model (IRG-Crowd) to recover the crowd information obscured by environmental factors. To ensure inference efficiency, we construct a reparameterizable student model (Rep-Crowd) that uses only a quarter of the parameters of the teacher model. Through multidimensional distillation, the student inherits the teacher’s robust feature representation without incurring additional restoration costs. Additionally, to address the scarcity of adverse weather datasets, we model the depth-of-field based on crowd density to construct non-uniform haze crowd datasets for MDKD training. Through extensive experiments, we find that our method outperforms state-of-the-art approaches for crowd counting and localization accuracy under diverse low-visibility conditions while significantly reducing the computational overhead. The code and datasets are available at https://github.com/lizhangray/MDKD-CC.
Audio-Visual event localization refers to identifying events that are visible and audible in videos using joint modeling of auditory and visual modalities to detect these events in temporal video segments. A challenge arises when the audio and visual contexts are inconsistent, and this information is clearly present (e.g., the on-screen visual shows a baby crying, while an off-screen female is speaking, resulting in conflicting information between the modalities). In such cases, both modalities exhibit high significance values, causing the model to misclassify background as an event. To address this, we propose a CLIP-based global context regulation method that leverages a pre-trained AudioCLIP encoder. This approach effectively regulates event-relevant scores through post-processing and performs well even with limited training data containing inconsistencies. We introduce a benchmark dataset annotated for inconsistent cases to facilitate robust evaluation. Experimental results demonstrate that our model outperforms existing methods and achieves state-of-the-art performance in event localization. These findings highlight the importance of regulating event overconfidence in multimodal inconsistency, contributing to more accurate event localization in real-world applications. Our code and dataset are available at: https://github.com/PangRAK/GCRN
Image restoration still poses significant challenges due to the complexity and diversity of real-world degradations. However, such degradations often manifest as spatially non-uniform, locally concentrated, and irregular patterns, which calls for a restoration model that can adapt its feature sampling to the underlying degradation geometry while remaining computationally efficient. In this paper, we design a novel image restoration framework built on a deformable restoration Transformer, termed DeRestormer. DeRestormer adapts to irregular but localized degradations via deformable attention, where a small set of sampling positions is learned to pre-filter and highlight key elements, so the model can better focus on informative regions and handle complex degradation patterns efficiently. In addition, we incorporate deformable convolutions at the bottleneck stage to dynamically adjust sampling locations across different scales, thus facilitating flexible multi-scale aggregation and effective integration of fine-grained details and global context. Extensive experiments on 13 datasets across 4 tasks demonstrate the effectiveness of DeRestormer, since it achieves state-of-the-art performance across a wide range of image restoration scenarios. The project and pretrained models of our work are available at https://github.com/Kingsley-Cyl/DeRestormer.
