Despite the impressive performance of Artificial Intelligence (AI) systems, their robustness remains elusive and constitutes a key issue that impedes large-scale adoption. Besides, robustness is interpreted differently across domains and contexts of AI. In this work, we systematically survey recent progress to provide a reconciled terminology of concepts around AI robustness. We introduce three taxonomies to organize and describe the literature both from a fundamental and applied point of view: (1) methods and approaches that address robustness in different phases of the machine learning pipeline; (2) methods improving robustness in specific model architectures, tasks, and systems; and in addition, (3) methodologies and insights around evaluating the robustness of AI systems, particularly the tradeoffs with other trustworthiness properties. Finally, we identify and discuss research gaps and opportunities and give an outlook on the field. We highlight the central role of humans in evaluating and enhancing AI robustness, considering the necessary knowledge they can provide, and discuss the need for better understanding practices and developing supportive tools in the future.

Transparency and explainability in image classification are essential for establishing trust in machine learning models and detecting biases and errors. State-of-the-art explainability methods generate saliency maps to show where a specific class is identified, without providing a detailed explanation of the model's decision process. Striving to address such a need, we introduce a post-hoc method that explains the entire feature extraction process of a Convolutional Neural Network. These explanations include a layer-wise representation of the features the model extracts from the input. Such features are represented as saliency maps generated by clustering and merging similar feature maps, to which we associate a weight derived by generalizing Grad-CAM for the proposed methodology. To further enhance these explanations, we include a set of textual labels collected through a gamified crowdsourcing activity and processed using NLP techniques and Sentence-BERT. Finally, we show an approach to generate global explanations by aggregating labels across multiple images.

In this article, we propose a hybrid, gamified, story-driven data collection approach to spark self-empathy, resurfacing people’s perceptions and feelings about past experiences. The game is designed around the well-known paradigm of an escape room and includes a physical board, some card decks, and a mobile application. We designed our concept around a very impactful experience that most people have experienced in recent years, namely the COVID-19 pandemic. The game aims to collect and understand people’s behaviour and feelings and to study and tackle the long-term effects of the pandemic. As the player plays through the game, they customize and escape from their lockdown room by completing statements and answering a series of questions that define their story. The decoration of the lockdown room and the storytelling-driven approach are targeted at sparking people’s emotions and self-empathy towards their past selves. Ultimately, the proposed approach was proven effective in collecting data about perceptions, opinions, and feelings related to the event.

This work focuses on human-AI interactions, employing a crowd-based methodology to collect and assess the reactions and perceptions of a human audience to a dialogue between a human and an artificial intelligent agent. The study is conducted through a live streaming platform where human streamers broadcast interviews to a custommade GPT voice interface. The questions extracted from the dialogues were categorized based on emotional and cognitive criteria. Our method covers thematic, emotional, and sentiment analyses of the comments platform users shared during the interview. This work aims to contribute to Human-Computer Interaction (HCI) and Human-Centered AI, emphasizing the need for a paradigm shift in AI research from focusing on technological development to considering its impact on human beings.

Digitally-supported participatory methods are often used in policy-making to develop inclusive policies by collecting and integrating citizen's opinions. However, these methods fail to capture the complexity and nuances in citizen's needs, i.e., citizens are generally unaware of other's needs, perspectives, and experiences. Consequently, policies developed with this underlying gap tend to overlook the alignment of multistakeholder perspectives, and design policies based on the optimization of high-level demographic features. In our contribution, we propose a method to enable citizens understand other's perspectives and calibrate their positions. First, we collected requirements and design principles to develop our approach by involving stakeholders and experts in policymaking in a series of workshops. Then, we conducted a crowdsourcing study with 420 participants to compare the effect of different text and images, on people's initial and final motivations and their willingness to change opinions. We observed that both influence participant's opinion change, however, the effect is more pronounced for textual modality. Finally, we discuss overarching implications of designing with empathy to mediate alignment of citizen's perspectives.

During the Covid-19 pandemic, research communities focused on collecting and understanding people's behaviours and feelings to study and tackle the pandemic indirect effects. Despite its consequences are slowly starting to fade away, such an interest is still alive. In this article, we propose a hybrid, gamified, story-driven data collection approach to spark self-empathy, hence resurfacing people's past feelings. The game is designed to include a physical board, decks of cards, and a digital application. As the player plays through the game, they customize and escape from their lockdown room by completing statements and answering a series of questions that define their story. The decoration of the lockdown room and the storytelling-driven approach are targeted at sparking people's emotions and self-empathy towards their past selves. Ultimately, the proposed approach was proven effective in sparking and collecting feelings, while a few improvements are still necessary.

Traditional approaches to data-informed policymaking are often tailored to specific contexts and lack strong citizen involvement and collaboration, which are required to design sustainable policies. We argue the importance of empathy-based methods in the policymaking domain given the successes in diverse settings, such as healthcare and education. In this paper, we introduce COCTEAU (Co-Creating The European Union), a novel framework built on the combination of empathy and gamification to create a tool aimed at strengthening interactions between citizens and policy-makers. We describe our design process and our concrete implementation, which has already undergone preliminary assessments with different stakeholders. Moreover, we briefly report pilot results from the assessment. Finally, we describe the structure and goals of our demonstration regarding the newfound formats and organizational aspects of academic conferences.

The spread of AI and black-box machine learning models made it necessary to explain their behavior. Consequently, the research field of Explainable AI was born. The main objective of an Explainable AI system is to be understood by a human as the final beneficiary of the model. In our research, we frame the explainability problem from the crowds point of view and engage both users and AI researchers through a gamified crowdsourcing framework. We research whether it's possible to improve the crowds understanding of black-box models and the quality of the crowdsourced content by engaging users in a set of gamified activities through a gamified crowdsourcing framework named EXP-Crowd. While users engage in such activities, AI researchers organize and share AI- and explainability-related knowledge to educate users. We present the preliminary design of a game with a purpose (G.W.A.P.) to collect features describing real-world entities which can be used for explainability purposes. Future works will concretise and improve the current design of the framework to cover specific explainability-related needs.

As the performance and complexity of machine learning models have grown significantly over the last years, there has been an increasing need to develop methodologies to describe their behaviour. Such a need has mainly arisen due to the widespread use of black-box models, i.e., high-performing models whose internal logic is challenging to describe and understand. Therefore, the machine learning and AI field is facing a new challenge: making models more explainable through appropriate techniques. The final goal of an explainability method is to faithfully describe the behaviour of a (black-box) model to users who can get a better understanding of its logic, thus increasing the trust and acceptance of the system. Unfortunately, state-of-the-art explainability approaches may not be enough to guarantee the full understandability of explanations from a human perspective. For this reason, human-in-the-loop methods have been widely employed to enhance and/or evaluate explanations of machine learning models. These approaches focus on collecting human knowledge that AI systems can then employ or involving humans to achieve their objectives (e.g., evaluating or improving the system). This article aims to present a literature overview on collecting and employing human knowledge to improve and evaluate the understandability of machine learning models through human-in-the-loop approaches. Furthermore, a discussion on the challenges, state-of-the-art, and future trends in explainability is also provided.

In recent years, new methods to engage citizens in deliberative processes of governments and institutions have been studied. Such methodologies have become a necessity to assure the efficacy and longevity of policies. Several tools and solutions have been proposed while trying to achieve such a goal. The dual problem to citizen engagement is how to provide policy-makers with useful and actionable insights stemming from those processes. In this paper, we propose a research featuring a method and implementation of a crowdsourcing and co-creation technique that can provide value to both citizens and policy-makers engaged in the policy-making process. Thanks to our methodology, policy-makers can design challenges for citizens to partake, cooperate and provide their input. We also propose a web-based tool that allow citizens to participate and produce content to support the policy-making processes through a gamified interface that focuses on emotional and vision-oriented content.

One year after the outbreak of the SARS-CoV-2, several vaccines have been successfully developed to prevent its spreading, and vaccine roll-out campaigns are taking place world-wide. However, an increasing number of individuals is still hesitant towards getting vaccinated, and this poses a serious threat to reaching herd immunity. We collect and analyze Italian online conversations about COVID-19 vaccines on Twitter. We define a hashtag-based semi-automatic approach to label large volumes of tweets as supporters or skeptical about the vaccine. We investigate the geographical, temporal and lexical distribution of data, and we train an accurate binary classifier that predicts the stance of tweets towards vaccines, i.e., it applies a “Pro-vax” or “No-vax” label. This classifi-cation approach can be used, in parallel with other affirmed techniques, to promptly detect and prevent the spread of negative and misleading messages about vaccines, ensuring higher rates of vaccine uptake.

We present VaccinItaly, a project which monitors Italian online conversations around vaccines, on Twitter and Facebook. We describe the ongoing data collection, which follows the SARS-CoV-2 vaccination campaign roll-out in Italy and we provide public access to the data collected. We show results from a preliminary analysis of the spread of low- and high-credibility news shared alongside vaccine-related conversations on both social media platforms. We also investigate the content of most popular YouTube videos and encounter several cases of harmful and misleading content about vaccines. Finally, we geolocate Twitter users who discuss vaccines and correlate their activity with open data statistics on vaccine uptake. We make up-to-date results available to the public through an interactive online dashboard associated with the project. The goal of our project is to gain further understanding of the interplay between the public discourse on online social media and the dynamics of vaccine uptake in the real world.

Over the last decades, communication between governments and citizens has become a remarkable problem. Governments’ decisions do not always match the visions of the citizens about the future. Achieving such alignment requires cooperation between communities and public institutions. Therefore, it’s essential to innovate governance and policymaking, developing new ways to harness the potential of public engagement and participatory foresight in complex governance decisions. This paper proposes a comprehensive framework that combines crowdsourcing and data analysis to improve the crowd’s collective engagement and contribution in policy-making decisions. Our approach brings together social networking, gamification, and data analysis practices to extract relevant and coordinated future visions concerning public policies. The framework is validated through two experiments with citizens and policy-making domain experts. The findings confirm the effectiveness of the framework principles and provide useful feedback for future development.