Recent advances in leveraging pre-trained Vision-Language Models (VLMs), such as CLIP, for open-set recognition (OSR) have mainly focused on learning textual prompts for known classes. Despite promising performance, textual-prompt-based methods still suffer from three limitations: insufficient characterization of low-level visual attributes, interference from image backgrounds, and distribution discrepancy between VLM pre-training data and target-task data. To address these issues, we develop a \underline{T}arget-\underline{A}ware \underline{D}ual \underline{P}rompt \underline{L}earning framework (TADPL). Building upon visual-prototype prompting as a complementary representation to textual prompts, TADPL introduces a new Target-Aware Enhancement (TAE) module that identifies target-aware known samples and constructs hard pseudo-unknown samples in the vicinity of known classes, thereby suppressing background interference and tightening known-class decision boundaries. A data-adaptive mechanism is further incorporated to improve the adaptability of the pre-trained VLM to the target data distribution. Moreover, we theoretically characterize the fusion gain induced by textual–visual prompt complementarity and establish a transfer bound relating pseudo-unknown separation to real-unknown separation. Extensive experiments on four standard OSR benchmarks demonstrate the effectiveness of TADPL, achieving average AUROC and OSCR scores of 97.4\% and 95.4\%, respectively, and consistently outperforming strong existing baselines.

Open-Set Recognition (OSR) aims to simultaneously classify samples of known classes and recognize samples of unknown classes, playing a crucial role in developing trustworthy models against errors on unforeseen inputs. Existing methods mainly tackle OSR from a single-granularity perspective, neglecting the inherently multi-granularity nature, which goes against a real-world intuition: even if a fine-grained Labrador is unknown, we can still recognize it as a Dog based on a coarser-grained label. Accordingly, we delve into exploring the internal potential of known classes by utilizing their hidden multi-granularity semantic structure for better closed-set training. Inspired by the Familiarity Hypothesis which reveals that the success of OSR methods mainly stems from detecting the absence of familiar features hidden within known classes, we develop Granularity-guIded FeAture FusioN STrategy (GIANT), where multi-granularity features of known classes are fused flexibly during training to better detect the absence of familiar closed-set features when unknown classes appear at test time. Specifically, to handle these multi-granularity features, we introduce two flexible feature fusion techniques, Intra-granularity Manifold Mixup (IM) to maintain semantic fidelity while ensuring diversity of representations at the same granularity, and Inter-granularity Concatenation (IC) to deepen familiarity with feature representations across various granularities. Theoretical analyses and heuristic experiments show that both components can enhance the recognition of networks. Extensive experimental results indicate our method achieves a new state-of-the-art.

In recent Open-set Recognition (OSR) community, a prevailing belief is that enhancing the discriminative boundaries of closed-set classes can improve the robustness of Deep Neural Networks (DNNs) against open data during testing. Typical studies validate this *implicitly* by empirical evidence, without a formalized understanding of *how DNNs help the closed-set features obtain more discriminative boundaries?* For this, we provide an answer from the Neural Collapse (NC) perspective: DNNs align the closed-set with a *Simplex Equiangular Tight Frame* (ETF) structure that has geometric and mathematical interpretability. Regrettably, although NC naturally occurs in DNNs, we discover that typical studies cannot guarantee the features being learned to strictly align with the ETF. Thus, we introduce a novel concept, Fixed ETF Template (FiT), which holds an ideal structure associated with closed-set classes. To force class means and classifier vectors to align with FiT, we further design a Dual ETF (DEF) loss involving two components. Specifically, *F*-DEF loss is designed to align class means with FiT strictly, yielding optimal inter-class separability. Meanwhile, we extend a dual form to classifier vectors, termed *C*-DEF loss, which guides class means and classifier vectors to satisfy self-duality. Our theoretical analysis proves the validity of the proposed approach, and extensive experiments demonstrate that DEF achieves comparable or superior results with reduced computational resources on standard OSR benchmarks.

Neural Collapse (NC) presents an elegant geometric structure that enables individual activations (features), class means and classifier (weights) vectors to reach optimal inter-class separability during the terminal phase of training on a balanced dataset. Once shifted to imbalanced classification, such an optimal structure of NC can be readily destroyed by the notorious minority collapse, where the classifier vectors corresponding to the minority classes are squeezed. In response, existing works mainly optimize classifiers in an effort to recover NC. However, we discover that this squeezing phenomenon is not only confined to classifier vectors but also occurs with class means. Consequently, reconstructing NC solely at the classifier aspect may be futile, as the class means remain compressed, leading to the violation of inherent self-duality in NC (i.e., class means and classifier vectors converge mutually) and incidentally, an unsatisfactory collapse of individual activations towards the corresponding class means. To shake off these dilemmas, we present a unified All-around Neural Collapse framework (AllNC), aiming to comprehensively restore NC across multiple aspects including individual activations, class means and classifier vectors. We thoroughly analyze its effectiveness and verify its performance on multiple benchmark datasets as state-of-the-art in both balanced and imbalanced settings.

Based on the success of large-scale visual foundation models like CLIP in various downstream tasks, this paper initially attempts to explore their impact on Long-Tailed Semi-Supervised Learning (LTSSL) by employing the foundation model with three strategies: Linear Probing (LP), Lightweight Fine-Tuning (LFT), and Full Fine-Tuning (FFT). Our analysis presents the following insights: i) Compared to LTSSL algorithms trained from scratch, FFT results in a decline in model performance, whereas LP and LFT, although boosting overall model performance, exhibit negligible benefits to tail classes. ii) LP produces numerous false pseudo-labels due to underlearned training data, while LFT can reduce the number of these false labels but becomes overconfident about them owing to biased fitting training data. This exacerbates the pseudo-labeled and classifier biases inherent in LTSSL, limiting performance improvement in the tail classes. With these insights, we propose a Unbiased Lightweight Fine-tuning strategy, ULFine, which mitigates the overconfidence via confidence-aware adaptive fitting of textual prototypes and counteracts the pseudo-labeled and classifier biases via complementary fusion of dual logits. Extensive experiments demonstrate that ULFine markedly decreases training costs by over ten times and substantially increases prediction accuracies compared to state-of-the-art methods.

Out-of-distribution detection (OOD) is a pivotal task for real-world applications that trains models to identify samples that are distributionally different from the in-distribution (ID) data during testing. Recent advances in AI, particularly Vision-Language Models (VLMs) like CLIP, have revolutionized OOD detection by shifting from traditional unimodal image detectors to multimodal image-text detectors. This shift has inspired extensive research; however, existing categorization schemes (e.g., few- or zero-shot types) still rely solely on the availability of ID images, adhering to a unimodal paradigm. To better align with CLIP's cross-modal nature, we propose a new categorization framework rooted in both image and text modalities. Specifically, we categorize existing methods based on how visual and textual information of OOD data is utilized within image + text modalities, and further divide them into four groups: OOD Images (i.e., outliers) Seen or Unseen, and OOD Texts (i.e., learnable vectors or class names) Known or Unknown, across two training strategies (i.e., train-free or training-required). More importantly, we discuss open problems in CLIP-like OOD detection and highlight promising directions for future research, including cross-domain integration, practical applications, and theoretical understanding.

Logit adjustment is an effective long-tailed visual recognition strategy to encourage a significant margin between rare and dominant labels. Existing methods typically employ the globally fixed label frequencies throughout the training to adjust margins. However, in practice, we observe that the local (in-batch) label frequencies change dynamically or even vanish for some classes (especially the tail classes) in batch-dependent training, which is inconsistent with global ones. Furthermore, our analyses reveal that the intra-class collinear samples actually do not contribute to the gradient update, but substantially increase the corresponding local label frequencies. Such contributions are spurious due to over-counting the label frequencies without contributing to the gradient. All of these will cause serious interference in precisely estimating local frequencies of the authentic contribution, leading to inauthentic margins. To simultaneously address the above issues, this paper innovatively proposes Dynamic Learnable Logit Adjustment (DLLA) loss to learn the local label frequencies within dynamic mini-batches precisely. Specifically, DLLA owns two complementary parts: 1) rank-metric eliminates spurious contributions from collinear samples by calculating the algebraic rank of the feature subspace in the mini-batch. 2) class-supplement ensures all classes appear in every mini-batch by inserting the corresponding learnable class prototype, for which we resort to neural collapse theory to make them align to the ideal regular simplex structure. Extensive experiments on standard benchmark datasets verify the effectiveness of our method.

In open-set recognition (OSR), a promising strategy is exploiting pseudo-unknown data outside given K known classes as an additional K+1-th class to explicitly model potential open space. However, treating unknown classes without distinction is unequal for them relative to known classes due to the category-agnostic and scale-agnostic of the unknowns. This inevitably not only disrupts the inherent distributions of unknown classes but also incurs both class-wise and instance-wise imbalances between known and unknown classes. Ideally, the OSR problem should model the whole class space as K+∞, but enumerating all unknowns is impractical. Since the core of OSR is to effectively model the boundaries of known classes, this means just focusing on the unknowns nearing the boundaries of targeted known classes seems sufficient. Thus, as a compromise, we convert the open classes from infinite to K, with a novel concept Target-Aware Universum (TAU) and propose a simple yet effective framework Dual Contrastive Learning with Target-Aware Universum (DCTAU). In details, guided by the targeted known classes, TAU automatically expands the unknown classes from the previous 1 to K, effectively alleviating the distribution disruption and the imbalance issues mentioned above. Then, a novel Dual Contrastive (DC) loss is designed, where all instances irrespective of known or TAU are considered as positives to contrast with their respective negatives. Experimental results indicate DCTAU sets a new state-of-the-art.