In the last decade, player and ball tracking data have increasingly been gathered and employed in various team sports, in particular invasion team sports, such as soccer, basketball, rugby, and American football. Additionally, the effectiveness of geometric analysis of formations using such tracking data has been demonstrated. Moreover, deep neural network-based methods for analyzing such tracking data have also been proposed. However, to the best of our knowledge, no research has been found that combines both, even though the benefits are seemingly apparent. In this study, against this background, we propose a formation analysis method for team sports using a deep graph neural network with geometric formation features. This method uses geometric formation features derived from a geometric formation analysis for tracking data. For initial validation, we also conduct an experiment applying this method to a soccer shooting scene classification problem with real soccer tracking data in Japan. Based on the results, we will discuss the method’s effectiveness.
Analyzing individual emotions during group conversation is crucial in developing intelligent agents capable of natural human-machine interaction. While reliable emotion recognition techniques depend on different modalities (text, audio, video), the inherent heterogeneity between these modalities and the dynamic cross-modal interactions influenced by an individual’s unique behavioral patterns make the task of emotion recognition very challenging. This difficulty is compounded in group settings, where the emotion and its temporal evolution are not only influenced by the individual but also by external contexts like audience reaction and context of the ongoing conversation. To meet this challenge, we propose a Multimodal Attention Network (MAN) that captures cross-modal interactions at various levels of spatial abstraction by jointly learning its interactive bunch of mode-specific Peripheral and Central networks. The proposed MAN "injects" cross-modal attention via its Peripheral keyvalue pairs within each layer of a mode-specific Central query network. The resulting cross-attended mode-specific descriptors are then combined using an Adaptive Fusion (AF) technique that enables the model to integrate the discriminative and complementary mode-specific data patterns within an instance-specific multimodal descriptor. Given a dialogue represented by a sequence of utterances, the proposed AMuSE (Adaptive Multimodal Analysis for Speaker Emotion) model condenses both spatial (within-mode and within-utterance) and temporal (across-mode and across-utterances in the sequence) features into two dense descriptors: speaker-level and utterance-level. This helps not only in delivering better classification performance (3−5% improvement in Weighted-F1 and 5−7% improvement in Accuracy) in large-scale public datasets (MELD and IEMOCAP) but also helps the users in understanding the reasoning behind each emotion prediction made by the model via its Multimodal Explainability Visualization module.
We have proposed a prototype fashion chatbot designed to retrieve similar outfit portraits and recommend reference users based on user-submitted outfit portraits. This chatbot uses a body part segmentation method to extract regions of body parts and calculates color feature values for the extracted body parts. This chatbot also employs an item segmentation method to extract fashion items present in the outfit portrait. Despite its capabilities, this chatbot showed limitations, notably retrieving portraits featuring incorrect types of outerwear. Furthermore, its retrieval process did not fully utilize the extracted results of the fashion item segmentation method. Thus, in this study, we aimed to address these issues by incorporating specific enhancements. We performed an initial validation experiment for the enhancements using outfit portraits posted on a Japanese fashion website.
Deep learning continues to reshape the potential of video action recognition, offering a robust framework to capture the basics of dynamic human movements. Despite the progress, efficient recognition of actions in videos remains a challenge. In our study, we introduce a comprehensive deep-learning model that effectively addresses this challenge. Specifically, our model is based on the spatiotemporal 3D convolutional neural network architecture with a multi-head attention mechanism to distinguish complex actions across video frames. The multi-layered architecture, including multiple linear layers and a multi-head attention mechanism, ensures both depth and precision in action classification. This paper elaborates throughout our study from raw video inputs to recognized actions, emphasizing the transformative potential of deep learning in enhancing action recognition capabilities. Our results on two benchmark datasets, namely UCF and HMDB, demonstrate that our proposed architecture improves the 3D ConvNet performance.
Video semantic segmentation has witnessed substantial advancements, largely due to the vast volume of labeled training samples. Nevertheless, domain adaptive video segmentation that adapts from a labeled source domain to an unlabeled target domain remains insufficiently delved into. In this paper, we propose Temporality-guided Masked Image Consistency (TgMIC), a simple yet effective approach that leverages the concept of Masked Image Modeling (MIM) to learn semantic features in the target domain. Unlike random masking strategy applied in traditional MIM, TgMIC introduces a novel temporality-guided masking strategy that samples the mask according to the distribution of optical flow, which facilitate the learning of spatial context relations in video sequence. Specifically, TgMIC masks the patches in vision transformers where the variance of optical flow is large, as these patches are known to contain noisy estimates of optical flow. In order to learn semantic information for video segmentation, TgMIC reconstructs the predictions of original frames from the masked frames. Comprehensive tests and detailed analysis on various public datasets show that our mechanism stands out, outpacing contemporaneous techniques, while ensuring no additional time costs.
