In the last decade, player and ball tracking data have increasingly been gathered and employed in various team sports, in particular invasion team sports, such as soccer, basketball, rugby, and American football. Additionally, the effectiveness of geometric analysis of formations using such tracking data has been demonstrated. Moreover, deep neural network-based methods for analyzing such tracking data have also been proposed. However, to the best of our knowledge, no research has been found that combines both, even though the benefits are seemingly apparent. In this study, against this background, we propose a formation analysis method for team sports using a deep graph neural network with geometric formation features. This method uses geometric formation features derived from a geometric formation analysis for tracking data. For initial validation, we also conduct an experiment applying this method to a soccer shooting scene classification problem with real soccer tracking data in Japan. Based on the results, we will discuss the method’s effectiveness.
Analyzing individual emotions during group conversation is crucial in developing intelligent agents capable of natural human-machine interaction. While reliable emotion recognition techniques depend on different modalities (text, audio, video), the inherent heterogeneity between these modalities and the dynamic cross-modal interactions influenced by an individual’s unique behavioral patterns make the task of emotion recognition very challenging. This difficulty is compounded in group settings, where the emotion and its temporal evolution are not only influenced by the individual but also by external contexts like audience reaction and context of the ongoing conversation. To meet this challenge, we propose a Multimodal Attention Network (MAN) that captures cross-modal interactions at various levels of spatial abstraction by jointly learning its interactive bunch of mode-specific Peripheral and Central networks. The proposed MAN "injects" cross-modal attention via its Peripheral keyvalue pairs within each layer of a mode-specific Central query network. The resulting cross-attended mode-specific descriptors are then combined using an Adaptive Fusion (AF) technique that enables the model to integrate the discriminative and complementary mode-specific data patterns within an instance-specific multimodal descriptor. Given a dialogue represented by a sequence of utterances, the proposed AMuSE (Adaptive Multimodal Analysis for Speaker Emotion) model condenses both spatial (within-mode and within-utterance) and temporal (across-mode and across-utterances in the sequence) features into two dense descriptors: speaker-level and utterance-level. This helps not only in delivering better classification performance (3−5% improvement in Weighted-F1 and 5−7% improvement in Accuracy) in large-scale public datasets (MELD and IEMOCAP) but also helps the users in understanding the reasoning behind each emotion prediction made by the model via its Multimodal Explainability Visualization module.
We have proposed a prototype fashion chatbot designed to retrieve similar outfit portraits and recommend reference users based on user-submitted outfit portraits. This chatbot uses a body part segmentation method to extract regions of body parts and calculates color feature values for the extracted body parts. This chatbot also employs an item segmentation method to extract fashion items present in the outfit portrait. Despite its capabilities, this chatbot showed limitations, notably retrieving portraits featuring incorrect types of outerwear. Furthermore, its retrieval process did not fully utilize the extracted results of the fashion item segmentation method. Thus, in this study, we aimed to address these issues by incorporating specific enhancements. We performed an initial validation experiment for the enhancements using outfit portraits posted on a Japanese fashion website.
Deep learning continues to reshape the potential of video action recognition, offering a robust framework to capture the basics of dynamic human movements. Despite the progress, efficient recognition of actions in videos remains a challenge. In our study, we introduce a comprehensive deep-learning model that effectively addresses this challenge. Specifically, our model is based on the spatiotemporal 3D convolutional neural network architecture with a multi-head attention mechanism to distinguish complex actions across video frames. The multi-layered architecture, including multiple linear layers and a multi-head attention mechanism, ensures both depth and precision in action classification. This paper elaborates throughout our study from raw video inputs to recognized actions, emphasizing the transformative potential of deep learning in enhancing action recognition capabilities. Our results on two benchmark datasets, namely UCF and HMDB, demonstrate that our proposed architecture improves the 3D ConvNet performance.
Video semantic segmentation has witnessed substantial advancements, largely due to the vast volume of labeled training samples. Nevertheless, domain adaptive video segmentation that adapts from a labeled source domain to an unlabeled target domain remains insufficiently delved into. In this paper, we propose Temporality-guided Masked Image Consistency (TgMIC), a simple yet effective approach that leverages the concept of Masked Image Modeling (MIM) to learn semantic features in the target domain. Unlike random masking strategy applied in traditional MIM, TgMIC introduces a novel temporality-guided masking strategy that samples the mask according to the distribution of optical flow, which facilitate the learning of spatial context relations in video sequence. Specifically, TgMIC masks the patches in vision transformers where the variance of optical flow is large, as these patches are known to contain noisy estimates of optical flow. In order to learn semantic information for video segmentation, TgMIC reconstructs the predictions of original frames from the masked frames. Comprehensive tests and detailed analysis on various public datasets show that our mechanism stands out, outpacing contemporaneous techniques, while ensuring no additional time costs.
This study demonstrates the ensemble approach to perform Human Action Recognition on the UT Dallas’ Multimodal Human Action Data, where number of actions by humans is 27. Our ensemble approach gained an accuracy of 0.821 on the validation data, a remarkable uplevelling as compared to the accuracy of baseline paper which is 0.672. The paper also shows the train-val performances of other models we experimented using only the Inertial and Skeleton dataset. The link to Github repository which holds to code can be found here, and the link to get the dataset can be found here.
This paper provides a brief overview on the innovative problem of devising and implementing big OLAP data cube compression algorithms in column-oriented Cloud/Edge data infrastructures, an emerging need for next-generation big data analytics systems.
In the last decade, data analysis and visualization have become increasingly essential in goal sports such as basketball, soccer, and rugby. Furthermore, in these goal sports, tracking systems have been widely used to obtain player and ball tracking data. Against this backdrop, several simulation-based methods have been proposed to generate informative data from player and ball tracking data of basketball matches, which are aimed to help fans, including viewers of match broadcasting or streaming, better understand plays and tactics. Among these methods, one involves a simulation- and optimization-based method for computing optimal dribbling routes to scoring areas in basketball. However, there was potential for further improvement. This study aims to improve this method by incorporating enhancements related to both a cost function for dribble routes and shootable areas. Additionally, this enhanced method is validated by using player and ball tracking data from the National Basketball Association (NBA).
Endowing AI with the ability of fine-art creation has been a long-standing goal for research community of multimedia. Indeed, recent image generation models have made to the point of shifting human art creation by generating visual image from text description. Although these AI generators are capable of producing end-to-end artworks with remarkable style and quality, they have not been able to imitate human in artistic creation by retrieving its painting process. Therefore, we focused on this challenging task of maximumly patterning human’s art creation by making text-to-painting image generation with fine-grained stroke by stroke predictions. Our proposal (SSG-Net) uses textual description as the input and further produces a quality artwork which not only matches the described scene but also restores the painting process with detailed sequential brushstroke predictions.
Computer vision has experienced remarkable advancements thanks to the rapid progress of deep learning techniques and the availability of extensive reference datasets. However, training models in the presence of limited annotated data remains a challenging task due to data dependency. To address this issue, few-shot learning has emerged as a promising solution, particularly in domains such as medical image analysis, in which accurate segmentation is crucial despite limited sample availability. Typically, a few-shot semantic segmentation model consists of a CNN encoder, a CNN decoder, and a simple classifier that must be learned in most of the existing approaches. However, an effective adaptation of a model considering all three components to the new class is very difficult since there are only a few support set images available. In this work, we focus on reducing meta-learning task complexity by concentrating only on the classifier, using a pre-trained encoder-decoder. Thus, we propose a new method, named Cross-Attention in Classifier Weight Transformer (Cross-CWT), that explores cross-attention to dynamically adjust the classifier’s weights to each query image. Experimental results show that the proposed approach can achieve competitive performance compared to other state-of-the-art methods with lesser training costs, making the few-shot semantic segmentation task much more tractable.
Dynamic graph learning has gained significant attention as it offers a powerful means to model intricate interactions among entities across various real-world and scientific domains. Notably, graphs serve as effective representations for diverse networks such as transportation, brain, social, and internet networks. Furthermore, the rapid advancements in machine learning have expanded the scope of dynamic graph applications beyond the aforementioned domains. In this paper, we present a review of lesser-explored applications of dynamic graph learning. This study revealed the potential of machine learning on dynamic graphs in addressing challenges across diverse domains, including those with limited levels of association with the field.
Graphs are a powerful representation tool in machine learning applications, with link prediction being a key task in graph learning. Temporal link prediction in dynamic networks is of particular interest due to its potential for solving complex scientific and real-world problems. Traditional approaches to temporal link prediction have focused on finding the aggregation of dynamics of the network as a unified output. In this study, we propose a novel perspective on temporal link prediction by defining nodes as Newtonian objects and incorporating the concept of velocity to predict network dynamics. By computing more specific dynamics of each node, rather than overall dynamics, we improve both accuracy and explainability in predicting future connections. We demonstrate the effectiveness of our approach using two datasets, including 17 years of co-authorship data from PubMed. Experimental results show that our temporal graph embedding dynamics approach improves downstream classification models’ ability to predict future collaboration efficacy in co-authorship networks by 17.34% (AUROC improvement relative to the baseline model). Furthermore, our approach offers an explainable layer over traditional approaches to address the temporal link prediction problem.
This work tackles the problem of bias in face matching algorithms. Face matching refers to the task of matching a low-resolution face image of a person with a high-resolution face image of the same person and has applications in security and personalization. Algorithmic bias is the difference in performance of an algorithm based on demographic descriptors of various users. Such bias can significantly reify and amplify societal biases and make certain advancements in technology benefit one section of the society while hurting the other. This work proposes a generative AI framework that can counter multiple kinds bias (e.g., gender bias and age bias) at the same time. The framework consists of two major components: a variational auto-encoder (VAE) that converts the images into their more generic underlying representation, and second, a neural network architecture that uses the above representations to undertake multi-label classification. A generative approach is useful in ensuring that the system learns to deal with the underlying (latent) structure of the data for better generalizability and bias reduction. The approach is tested over a public image dataset and found to be effective at reducing bias while maintaining high accuracy.
In this study, we propose a method to compute optimal dribbling routes to areas from which the ball handler can shoot. We also equip a visualization tool with a function for visualizing the optimal dribbling routes computed with the proposed method. In addition, we validate this visualization function by using real location data of the players and the ball.
Semi-supervised learning (SSL) is a classic missing label problem. Existing SSL algorithms always rely on the basic assumption, label Missing Completely At Random (MCAR), where both labeled and unlabeled data share the same class distribution. Compared to MCAR, the label Missing Not At Random (MNAR) problem is more realistic. In MNAR, the labeled and unlabeled data have different class distributions resulting in biased label imputation, which leads to the performance degradation of SSL models. Existing SSL algorithms can hardly perform well on tail classes (the classes with few training examples) in MNAR setting, since the pseudo-labels learned from unlabeled data tend to be biased toward head classes (the classes with a large number of training examples). To alleviate this issue, we propose a Class-Aware Pseudo Labeling (CAPL) for Non-random Missing Labels in SSL, which utilizes the unlabeled data by dynamically adjusting the threshold for selecting pseudo-labels. Under various MNAR settings, our method achieves up to 15.0% overall accuracy gain upon FixMatch in CIFAR-10 compared with existing baselines.
Sharing information on social media has become a part of people's daily lives. However, without centralized management on user generated contents, increasing amount of rumors and misinformation are being spread in social media. On the one hand, most of the social platforms debunk rumors with manual verification by fact checking organizations, which are very inefficient. On the other hand, since it's common for rumors to contain inconsistent images and texts, it would be useful if we could compare the semantics between multimodal contents in the same post for rumor detection. In this paper, we propose to check multimodal content consistency with transformers and self-attention-based Bi-GRU networks for rumor detection. Firstly, image semantic contents are extracted by image captioning module to generate captions. Then, the generated captions are semantically compared with texts using transformers for veracity assessment. Finally, Multi-cell bi-directional Recurrent Neural Networks (Bi-RNNs) with self-attention mechanism are used to find word dependency and learn the most important features for rumor detection. From the experimental results on tweets, the best F1-score of 0.92 can be obtained for our proposed approach to multimodal veracity assessment. This shows the potential of our proposed method in rumor detection. Further investigation is needed to verify the performance using different multimodal features.
Today's Web aims to increase engagement with the user, his or her emotions and behaviors. Contemplating the emotional sphere is potentially very useful when it comes to Web sites: understanding how the user feels can help the service provider understand his or her needs and improve the service he or she offers. This project aims to devise a behavioral feature extraction system to create an interactive system that can provide a personalized experience and maximize user satisfaction. The system contemplates behavioral analysis techniques using Machine Learning, Recommender System and Emotion Recognition algorithms. In the experimental phase, users tested the system through Expressing Mixed Emotions, SUS and SUPR-Q tasks and questionnaires with the aim of evaluating the User Experience and The Usability of the system.
This paper focuses the attention on the problem of supporting multidimensional big data analytics tools over Big Web Knowledge Bases. These big data repositories are generated from web systems built on top of big datasets, as in several real-life examples such as web interfaces to social networks, e-government systems, smart city tools (e.g., for weather forecasting), and so forth. The problem of extracting actionable knowledge insights from big web knowledge bases is a relevant challenge at now. In line with this emerging research field, the paper proposes models, issues and research trends in the context of multidimensional big data analytics over big web knowledge bases, along with a reference architecture that impersonates this paradigm.
Image segmentation is the foundation of several computer vision tasks, where pixel-wise knowledge is a pre-requisite for achieving the desired target. Deep learning has shown promising performance in supervised image segmentation. However, supervised segmentation algorithms require a massive amount of data annotated at a pixel level, thus limiting their applicability and scalability. Therefore, there is a need to invest in unsupervised learning for segmentation. This work presents an improved version of an unsupervised Convolutional Neural Network (CNN) based algorithm that uses a constant weight factor to balance between the segmentation criteria of feature similarity and spatial continuity, and it requires continuous manual adjustment of parameters depending on the degree of detail in the image and the dataset. In contrast, we propose a novel dynamic weighting scheme that leads to a flexible update of the parameters and an automatic tuning of the balancing weight between the two criteria above to bring out the details in the images in a genuinely unsupervised manner. We present quantitative and qualitative results on four datasets, which show that the proposed scheme outperforms the current unsupervised segmentation approaches without requiring manual adjustment.
