Recently, the advancement of deep learning has led to considerable breakthroughs in the automated detection of ActionUnits (AUs). Nevertheless, this field is still faced with various challenges, including limited subjects in shared datasetsand the difficulty of collecting AU data as domain knowledge required for annotating AUs. These issues make it arduousfor the model to generalize across different subjects and attain satisfactory performance on all AUs. To address thesechallenges, we propose two methods, namely AU-specific Querying (AQ) and Feedback Querying (FQ). AQ learns theglobal semantics of a particular AU, while FQ provides local structure information related to the AU semantic vector.The combination of these two operations enables the model to leverage both local features and global semantics of AUs.Moreover, Feedback Querying exhibits strong extensibility, which has led us to propose Cross-subject Querying (CQ).This method learns a subject-independent feature representation for each AU, resulting in improved generalizationability across different subjects. We demonstrate the effectiveness of our methods through visual presentations andablation analysis. By combining all the strategies, our proposed AQ-CSL becomes the state-of-the-art model on theDISFA and BP4D datasets.

In recent years, patch-based approaches have shown promise in few-shot learning, with further improvements observed through the use of self-supervised learning. However, we observe that the mainstream object-oriented approach focuses mainly on the salient part of the subject and ignores the non-annotated part of the image. Based on the assumption that any patch of the image is beneficial to learning, we present an end-to-end learning framework, which reconsiders the whole image from a multi-level perspective. The learning of annotated subjects involves Direct Patch Learning (DPL) to promote balanced learning of different features and Gaussian Mixup (GMIX) to provide extra mixed patch-level labels. As for the non-annotated part, we utilize a cascading token selection strategy along with self-supervised learning to better utilize knowledge in the background in the current context by learning the consistent representation of different views from the same image. Finally, in inductive few-shot learning, our method outperforms many previous methods and achieves new state-of-the-art performance. Furthermore, it provides an insight that non-annotated parts are also favorable for few-shot learning. As an ablation study, the effectiveness of each designed component is verified.

In recent years, patch-based approaches have shown promise in few-shot learning, with further improvements observed through the use of self-supervised learning. However, we observe that the mainstream object-oriented approach focuses mainly on the salient part of the subject and also ignores the non-annotated part of the image. Based on the assumption that any patch of the image is beneficial to learning, we present an end-to-end learning framework, which reconsiders the whole image from a multi-level perspective. The learning of annotated subjects involves Direct Patch Learning (DPL) to promote balanced learning of different features, and Gaussian Mixup (GMIX) to provide extra mixed patch-level labels. As for the non-annotated part, we utilize a cascading token selection strategy along with self-supervised learning to better utilize knowledge in the background in the current context by learning the consistent representation of different views from the same image. Finally, in inductive few-shot learning, our method outperforms many previous methods and achieves new state-of-the-art performance. Furthermore, it provides an insight that non-annotated parts still is favorable for few-shot learning. As an ablation study, the effectiveness of each designed component is verified and the mechanism of how our method outperforms the baseline is shown both quantitatively and visually.

Facial Action Units (AUs) are of great significance in communication. Automatic AU detection can improve the understanding of psychological conditions and emotional status. Recently, several deep learning methods have been proposed to detect AUs automatically. However, several challenges, such as poor extraction of fine-grained and robust local AUs information, model overfitting on person-specific features, as well as the limitation of datasets with wrong labels, remain to be addressed. In this paper, we propose a joint strategy called PIAP-DF to solve these problems, which involves 1) a multi-stage Pixel-Interested learning method with pixel-level attention for each AU; 2) an Anti Person-Specific method aiming to eliminate features associated with any individual as much as possible; 3) a semi-supervised learning method with Discrete Feedback, designed to effectively utilize unlabeled data and mitigate the negative impacts of wrong labels. Experimental results on the two popular AU detection datasets BP4D and DISFA prove that PIAP-DF can be the new state-of-the-art method. Compared with the current best method, PIAP-DF improves the average F1 score by 3.2% on BP4D and by 0.5% on DISFA. All modules of PIAP-DF can be easily removed after training to obtain a lightweight model for practical application.