Europe is one of the most forest-rich regions in the world, with forestry mainly based on the management of coniferous trees. However, the spruce forest ecosystem is vulnerable to several disturbance agents. In particular, bark beetle outbreaks have been the scourge of spruce trees in the last decade, and they are expected to further intensify due to climate change, with significant adverse effects on forest ecosystems. Hence, the monitoring of forest disturbances caused by the rapidly escalating bark beetle outbreaks represents a significant ecological and forestry challenge. This monitoring is traditionally performed by foresters during field surveys. On the other hand, open Sentinel-2 images, available with the Copernicus mission and processed with sophisticated deep learning techniques, have been recently established as an alternative to field surveys performed by foresters to monitor various environmental phenomena such as bark beetle outbreaks. In particular, several deep learning approaches have been recently proposed to map bark beetle tree dieback using Sentinel-2 images of forests. However, the current effectiveness of deep learning approaches, as a means to monitor bark beetle outbreaks in Sentinel-2 data is often limited by the reduced availability of ground truth information to supervise semantic segmentation models for this specific downstream task. In this study, we propose ULISSE, a deep learning semantic segmentation methodology for mapping forest tree dieback caused by bark beetle outbreak disturbances using Sentinel-2 image time series. ULISSE leverages a U-Net-like architecture with a multi-temporal encoder specifically designed to handle Sentinel-2 image time series. The framework integrates vision encoders pretrained on a large volume of Sentinel-2 images for land-cover classification. To capitalize on the representational capabilities of these pretrained encoders, we employ a Parameter-Efficient Fine-Tuning (PEFT) mechanism that adapts the multi-temporal encoder to our downstream segmentation task. This approach enables ULISSE to achieve high accuracy even with a limited amount of labeled data available at training time. Experimental results demonstrate the effectiveness of the proposed methodology across two case studies on mapping bark beetle disturbances in the Czech Republic and Romania study areas.

In recent years, Predictive Process Monitoring (PPM) has evolved at the intersection of process mining, machine learning, and data science, as organizations seek to anticipate the future course of ongoing processes. Traditional PPM mainly relies on structured event log data, but many real-world scenarios generate richer information, including text, images, audio, and video. Multimodal Predictive Process Monitoring (MM-PPM) addresses this rich data scenario by integrating complementary knowledge from heterogeneous modalities through modality-specific representations and information fusion techniques. The growing digitization of healthcare systems, combined with advances in Artificial Intelligence (AI), has accelerated AI-based PPM for analyzing sequences of clinical events, supporting decision-making, enabling personalized care, and improving clinical facility management. Given these characteristics, clinical pathways represent an ideal domain for experimenting with MM-PPM, as they may naturally involve diverse modalities such as structured records, free-text notes, or medical images. To handle multimodal information available with clinical pathways, we introduce MEDUSA, an MM-PPM approach for outcome prediction, which jointly processes medical image information coupled with the storytelling of structural records and text notes collected during the clinical pathway of a patient until the acquisition of the considered image. The evaluation of MEDUSA is done in a COVID-19 case study, to assess the performance of the proposed approach and explain how specific information within each modality influences the decisions of the predictive model.

Deepfake detectors are typically trained on large sets of pristine and generated images, resulting in limited generalization capacity; they excel at identifying deepfakes created through methods encountered during training but struggle with those generated by unknown techniques. This paper introduces a learning approach aimed at significantly enhancing the generalization capabilities of deepfake detectors. Our method takes inspiration from the unique "fingerprints" that image generation processes consistently introduce into the frequency domain. These fingerprints manifest as structured and distinctly recognizable frequency patterns. We propose to train detectors using only pristine images injecting in part of them crafted frequency patterns, simulating the effects of various deepfake generation techniques without being specific to any. These synthetic patterns are based on generic shapes, grids, or auras. We evaluated our approach using diverse architectures across 25 different generation methods. The models trained with our approach were able to perform state-ofthe-art deepfake detection, demonstrating also superior generalization capabilities in comparison with previous methods. Indeed, they are untied to any specific generation technique and can effectively identify deepfakes regardless of how they were made. We published the code online1 to use the proposed approach and reproduce the presented experiments.

Android OS is today the most used Operating System for mobile devices. However, it is susceptible to several malware attacks that may seriously compromise the privacy and security of individuals and organizations. This paper proposes an approach based on a static analysis of decompiled Android PacKages (APKs) to extract critical APIs and detect Android malware. The main contributions lie in the adoption of a graph-based data engineering schema to represent APIs taken from the Function Call Graphs of decompiled APKs and the formulation of a graph-based deep learning approach for explainable malware detection. In particular, the proposed approach, named ANAKIN, implements a Graph Neural Network (GNN) for binary classification (malware versus goodware), and integrates GNNExplainer algorithm to disclose how specific API classes and control-flow edges between API calls influence malware alerts. The proposed approach was evaluated by considering 26,527 Android APKs. The results of an extensive and in-depth evaluation show that the presented GNN model achieves higher accuracy than deep neural models trained with traditional API call sequence representations and publicly available related methods. On the other hand, it produces decision explanations that yield interesting insights into the malicious patterns of APKs and support root cause analysis of missed malware alarms.

We present GraphCLIP, a novel contrastive learning framework for multimodal artwork classification that integrates visual and contextual information to improve predictive accuracy and interpretability. Traditional computer vision methods often fall short in visual arts, where context is crucial. GraphCLIP leverages image data and a Knowledge Graph to extract features from both perspectives. Evaluated on the ArtGraph dataset, with over 100,000 artworks in 32 styles and 18 genres, GraphCLIP outperforms existing models in single-task (up to +8% in F1-score) and multi-task settings (up to +6%), demonstrating robustness even with unseen classes. Additionally, visual and contextual qualitative explanations enhance model transparency. The versatility of GraphCLIP extends beyond art classification: its methodology can be adapted to other domains where integrating diverse data types is essential. (The code is publicly available at: https://github.com/CILAB-ArtGraph/graphclip.git.)

Huge spruce forest areas have been damaged by massive bark beetle outbreaks across Europe during the past few years. Hence, forest health management requires large-scale inventory of bark beetle outbreaks to plan actions for promptly mitigating forest tree dieback. Deep learning techniques have recently achieved amazing results in imagery semantic segmentation tasks by dominating the recent research for mapping bark beetle outbreaks in Sentinel-2 images of forest areas. In addition, due to the impressive performance of Large Language Models (LLMs) in natural language understanding and generation tasks, LLMs have started attracting attention in multiple fields. In this paper, we describe GANDALF: an approach that leverages the potential of LLMs for mapping bark beetle outbreaks in Sentinel-2 images of forest areas. Specifically, we take advantage of the rich context of textual data to transform Sentinel-2 images in smart data ready for boosting accurate semantic segmentation modeling. We use a foundation LLM model to account for the text encoding of the spectral-spatial imagery context information. We fine-tune the LLM model to perform the semantic segmentation of forest images and use the Integrated Gradients (IG) algorithm to explain how each spectral-spatial information has an effect on the bark beetle outbreak detection. We assess the effectiveness of the proposed approach in a case study regarding bark beetle outbreaks in Sentinel-2 images of forest scenes in Czech Republic.

Few-shot semantic segmentation aims to segment objects from previously unseen classes using only a limited number of labeled examples. In this paper, we introduce Label Anything, a novel transformer-based architecture designed for multi-prompt, multi-way few-shot semantic segmentation. Our approach leverages diverse visual prompts—points, bounding boxes, and masks—to create a highly flexible and generalizable framework that significantly reduces annotation burden while maintaining high accuracy. Label Anything makes three key contributions: (i) we introduce a new task formulation that relaxes conventional few-shot segmentation constraints by supporting various types of prompts, multi-class classification, and enabling multiple prompts within a single image; (ii) we propose a novel architecture based on transformers and attention mechanisms; and (iii) we design a versatile training procedure allowing our model to operate seamlessly across different N-way K-shot and prompt-type configurations with a single trained model. Our extensive experimental evaluation on the widely used COCO-20i benchmark demonstrates that Label Anything achieves state-of-the-art performance among existing multi-way few-shot segmentation methods, while significantly outperforming leading single-class models when evaluated in multi-class settings. Code and trained models are available at https://github.com/pasqualedem/LabelAnything.

Video action anticipation is a specific field within computer vision that diverges from action recognition, requiring the prediction of future actions through the analysis of historical video sequences. This paper unveils an innovative model designed to overcome the limitations of existing solutions by amalgamating recurrent and attention mechanisms, taking cues from the principles of object tracking. Notably, our model leverages prior anticipation results, enabling a nuanced interpretation of semantic transitions between actions and recognizing the uncertainty inherent in predicting future events. This strategy strikes a balance between computational efficiency and judicious data utilization, challenging the assumptions prevalent in current transformer models and thereby underlining its practicality for real-world applications. Distinctively, our model discerns temporal connection from abstract concepts in a way that mirrors human reasoning and adopts a recurrent structure to thoroughly capture video context. Extensive experiments conducted on EPIC-Kitchens-100, EPIC-Kitchens-55, and EGTEA Gaze+ confirm the superior performance of our proposed model and efficiency compared to established transformer architectures. Remarkably, it surpasses most multi-modality models using only RGB visual inputs, showcasing its exceptional generalization capabilities across a variety of unseen test sets.

Along with the nearing completion of the Square Kilometre Array (SKA), comes an increasing demand for accurate and reliable automated solutions to extract valuable information from the vast amount of data it will allow acquiring. Automated source finding is a particularly important task in this context, as it enables the detection and classification of astronomical objects. Deep-learning-based object detection and semantic segmentation models have proven to be suitable for this purpose. However, training such deep networks requires a high volume of labeled data, which is not trivial to obtain in the context of radio astronomy. Since data needs to be manually labeled by experts, this process is not scalable to large dataset sizes, limiting the possibilities of leveraging deep networks to address several tasks. In this work, we propose RADiff, a generative approach based on conditional diffusion models trained over an annotated radio dataset to generate synthetic images, containing radio sources of different morphologies, to augment existing datasets and reduce the problems caused by class imbalances. We also show that it is possible to generate fully-synthetic image-annotation pairs to automatically augment any annotated dataset. We evaluate the effectiveness of this approach by training a semantic segmentation model on a real dataset augmented in two ways: 1) using synthetic images obtained from real masks, and 2) generating images from synthetic semantic masks. We show an improvement in performance when applying augmentation, gaining up to 18% in performance when using real masks and 4% when augmenting with synthetic masks. Finally, we employ this model to generate large-scale radio maps with the objective of simulating Data Challenges.

Deepfake detectors are typically trained on large sets of pristine and generated images, resulting in limited generalization capacity; they excel at identifying deepfakes created through methods encountered during training but struggle with those generated by unknown techniques. This paper introduces a learning approach aimed at significantly enhancing the generalization capabilities of deepfake detectors. Our method takes inspiration from the unique "fingerprints" that image generation processes consistently introduce into the frequency domain. These fingerprints manifest as structured and distinctly recognizable frequency patterns. We propose to train detectors using only pristine images injecting in part of them crafted frequency patterns, simulating the effects of various deepfake generation techniques without being specific to any. These synthetic patterns are based on generic shapes, grids, or auras. We evaluated our approach using diverse architectures across 25 different generation methods. The models trained with our approach were able to perform state-of-the-art deepfake detection, demonstrating also superior generalization capabilities in comparison with previous methods. Indeed, they are untied to any specific generation technique and can effectively identify deepfakes regardless of how they were made.

In this paper, we propose ViTO, a novel knowledge distillation strategy that aims to convert a CNN model into a transformer-based counterpart that incorporates the advantages of transformers while retaining or improving its inductive bias. Our approach is based on a two-level transformer architecture that includes an inner model for learning visual representations and an outer model that aims to match the teacher’s predictions through autoregression. Specifically, given an image in a batch, the outer model classifies the image by using, in addition to the image’s visual properties, also the predictions it has made on images previously seen within the same batch. The effect of this strategy is to allow the transformer to estimate self- and cross-attention across all input batch images to learn autoregressively intra-class and inter-class correlations.We experimentally validate ViTO on several standard benchmarks obtaining better performance than existing knowledge distillation strategies on transformers. Furthermore, our distilled transformer-based model shows better robustness properties than standard vision transformers, demonstrating the effectiveness of our proposed distillation strategy.

Deep-learning models for 3D point cloud semantic segmentation exhibit limited generalization capabilities when trained and tested on data captured with different sensors or in varying environments due to domain shift. Domain adaptation methods can be employed to mitigate this domain shift, for instance, by simulating sensor noise, developing domain-agnostic generators, or training point cloud completion networks. Often, these methods are tailored for range view maps or necessitate multi-modal input. In contrast, domain adaptation in the image domain can be executed through sample mixing, which emphasizes input data manipulation rather than employing distinct adaptation modules. In this study, we introduce compositional semantic mixing for point cloud domain adaptation, representing the first unsupervised domain adaptation technique for point cloud segmentation based on semantic and geometric sample mixing. We present a two-branch symmetric network architecture capable of concurrently processing point clouds from a source domain (e.g. synthetic) and point clouds from a target domain (e.g. real-world). Each branch operates within one domain by integrating selected data fragments from the other domain and utilizing semantic information derived from source labels and target (pseudo) labels. Additionally, our method can leverage a limited number of human point-level annotations (semi-supervised) to further enhance performance. We assess our approach in both synthetic-to-real and real-to-real scenarios using LiDAR datasets and demonstrate that it significantly outperforms state-of-the-art methods in both unsupervised and semi-supervised settings.

Source finding is one of the most challenging tasks in upcoming radio continuum surveys with SKA precursors, such as the Evolutionary Map of the Universe (EMU) survey of the Australian SKA Pathfinder (ASKAP) telescope. In this work, we explore the application of several deep learning methods, based both on classical CNNs and on more recent transformer architectures to perform object detection. All the models were trained using ASKAP data, taken during the Early Science phase, and previous radio survey data. Results in terms of Precision, Recall, and F1-Score are promising. We evaluate several training strategies and model architectures to determine which one fits the case study best. Finally, we exploited High Performance Computing to run our experiments, to evaluate the improvement of a distributed training process, running on multiple computing nodes, compared to a single-node, single-process approach.

This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources, containing both human-annotated and web-collected captions. Large-scale datasets with noisy image-text pairs, indeed, provide a sub-optimal source of supervision because of their low-quality descriptive style, while human-annotated datasets are cleaner but smaller in scale. To get the best of both worlds, we propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component. The proposed model avoids the need of object detectors, is trained with a single objective of prompt language modeling, and can replicate the style of human-collected captions while training on sources with different input styles. Experimentally, the model shows a strong capability of recognizing real-world concepts and producing high-quality captions. Extensive experiments are performed on different image captioning datasets, including CC3M, nocaps, and the competitive COCO dataset, where our model consistently outperforms baselines and state-of-the-art approaches.

In recent years, deep learning has been successfully applied in various scientific domains. Following these promising results and performances, it has recently also started being evaluated in the domain of radio astronomy. In particular, since radio astronomy is entering the Big Data era, with the advent of the largest telescope in the world - the Square Kilometre Array (SKA), the task of automatic object detection and instance segmentation is crucial for source finding and analysis. In this work, we explore the performance of the most affirmed deep learning approaches, applied to astronomical images obtained by radio interferometric instrumentation, to solve the task of automatic source detection. This is carried out by applying models designed to accomplish two different kinds of tasks: object detection and semantic segmentation. The goal is to provide an overview of existing techniques, in terms of prediction performance and computational efficiency, to scientists in the astrophysics community who would like to employ machine learning in their research.

Source finding is one of the most challenging tasks in upcoming radio continuum surveys with SKA precursors, such as the Evolutionary Map of the Universe (EMU) survey of the Australian SKA Pathfinder (ASKAP) telescope. The resolution, sensitivity, and sky coverage of such surveys is unprecedented, requiring new features and improvements to be made in existing source finders. Among them, reducing the false detection rate, particularly in the Galactic plane, and the ability to associate multiple disjoint islands into physical objects. To bridge this gap, we developed a new source finder, based on the Mask R-CNN object detection framework, capable of both detecting and classifying compact, extended, spurious, and poorly imaged sources in radio continuum images. The model was trained using ASKAP EMU data, observed during the Early Science and pilot survey phase, and previous radio survey data, taken with the VLA and ATCA telescopes. On the test sample, the final model achieves an overall detection completeness above 85%, a reliability of ∼65%, and a classification precision/recall above 90%. Results obtained for all source classes are reported and discussed.

As the development of deep learning (DL) techniques has progressed, the creation of convincing synthetic media, known as deepfakes, has become increasingly easy, raising significant concern about the use of these videos to spread false information and potentially manipulate public opinion. In recent years, deep neural networks, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have been used for deepfake detection systems, exploiting the inconsistencies and the artifacts introduced by generation algorithms. Taking into account the main limitation of fake videos to realistically reproduce the natural human emotion patterns, in this paper, we present FEAD-D, a publicly available tool for deepfake detection performing facial expression analysis. Our system exploits data from the DeepFake Detection Challenge (DFDC) and consists of a model based on bidirectional Long Short-Term Memory (BiLSTM) capable of detecting a fake video in about two minutes with an overall accuracy of 84.29% on the test set (i.e. comparable with the current state-of-the-art, while consisting of fewer parameters), showing that emotional analysis can be used as a robust and reliable method for deepfake detection.