This study addresses the notable gap in research on detecting private-sensitive content within Norwegian social media by creating and annotating a dataset, tailored specifically to capture the linguistic and cultural nuances of Norwegian social media discourse. Utilizing Reddit as a primary data source, entries were compiled and cleaned, resulting in a comprehensive dataset of 4482 rows. Our research methodology encompassed evaluating a variety of computational models—including machine learning, deep learning, and transformers—to assess their effectiveness in identifying sensitive content. Among these, the NB BERT-based classifier emerged as the proficient, showcasing accuracy and F-1 score. This classifier demonstrated remarkable effectiveness, achieving an accuracy of 82.75% and an F1-score of 82.39%, underscoring its adeptness at navigating the complexities of privacy-sensitive content detection in Norwegian social media. This endeavor not only paves the way for enhanced privacy-sensitive content detection in Norwegian social media but also sets a precedent for future research in the domain, emphasizing the critical role of tailored datasets in advancing the field.
LambdaMART, a potent black-box Learning-to-Rank (LTR) model, has been shown to outperform neural network models across tabular ranking benchmark datasets. However, its lack of transparency challenges its application in many real-world domains. Local list-wise explanation techniques provide scores that explain the importance of the features in a list of documents associated with a query to the prediction of black-box LTR models. This study investigates which list-wise explanation techniques provide the most faithful explanations for LambdaMART models. Several local explanation techniques are evaluated for this, i.e., Greedy Score, RankLIME, EXS, LIRME, LIME, and SHAP. Moreover, a non-LTR explanation technique is applied, called Permutation Importance (PMI) to obtain list-wise explanations of LambdaMART. The techniques are compared based on eight evaluation metrics, i.e., Consistency, Completeness, Validity, Fidelity, ExplainNCDG@10, (In)fidelity, Ground Truth, and Feature Frequency similarity. The evaluation is performed on three benchmark datasets: Yahoo, Microsoft Bing Search (MSLR-WEB10K), and LETOR 4 (MQ2008), along with a synthetic dataset. The experimental results show that no single explanation technique is faithful across all datasets and evaluation metrics. Moreover, the explanation techniques tend to be faithful for different subsets of the evaluation metrics; for example, RankLIME out-performs other explanation techniques with respect to Fidelity and ExplainNCDG, while PMI provides the most faithful explanations with respect to Validity and Completeness. Moreover, we show that explanation sample size and the normalization of feature importance scores in explanations can largely affect the faithfulness of explanation techniques across all datasets.
The training success of deep learning is known to depend on the initial statistics of neural network parameters. Various strategies have been developed to determine suitable mean and standard deviation for weight distributions based on network architecture. However, during training, weights often diverge from their initial scale. This paper introduces the novel concept of weight rescaling, which enforces weights to remain within their initial regime throughout the training process. It is demonstrated that weight rescaling serves as an effective regularization method, reducing overfitting and stabilizing training while improving neural network performance. The approach rescales weight vector magnitudes to match the initialization methods’ conditions without altering their direction. It exhibits minimal memory usage, is lightweight on computational resources and demonstrates comparable results to weight decay, but without introducing additional hyperparameters as it leverages architectural information. Empirical testing shows improved performance across various architectures, even when combined with additional regularization methods like dropout in AlexNet and batch normalization in ResNet-50. The effectiveness of weight rescaling is further supported by a thorough statistical evaluation.
Neural Ranking Models have shown state-of-the-art performance in Learning-To-Rank (LTR) tasks. However, they are considered black-box models. Understanding the logic behind the predictions of such black-box models is paramount for their adaptability in the real-world and high-stake decision-making domains. Local explanation techniques can help us understand the importance of features in the dataset relative to the predicted output of these black-box models. This study investigates new adaptations of Local Interpretable Model-Agnostic Explanation (LIME) explanation for explaining Neural ranking models. To evaluate our proposed explanation, we explain Neural GAM models. Since these models are intrinsically interpretable Neural Ranking Models, we can directly extract their ground truth importance scores. We show that our explanation of Neural GAM models is more faithful than explanation techniques developed for LTR applications such as LIRME and EXS and non-LTR explanation techniques for regression models such as LIME and KernelSHAP using measures such as Rank Biased Overlap (RBO) and Overlap AUC. Our analysis is performed on the Yahoo! Learning-To-Rank Challenge dataset.
Synthetic data promises to address several challenges in training machine learning models, such as data scarcity, privacy concerns, and efforts for data collection and annotation. In order to actually benefit from synthetic data, its utility for the intended purpose has to be ensured and, ideally, estimated before it is used to produce possibly poorly performing models. Population fidelity metrics are potential candidates to provide such an estimation. However, evidence of how well they are suited as estimators of the utility of synthetic data is scarce.In this study, we present the results of an experiment in which we investigated whether population fidelity as measured with nine different metrics correlates with the predictive performance of classification models trained on synthetic data.Cluster Analysis and Cross-Classification show the most consistent results w.r.t. correlation with F1-performance but do not exceed moderate levels.The degree of correlation, and hence the potential suitability for estimating utility, varies considerably across the inspected datasets. Overall, the results suggest that the inspected population fidelity metrics are not a reliable and accurate tool to estimate the utility of synthetic training data for classification tasks. They may be precise enough though to indicate trends for different synthetic datasets based on the same original data.Further research should shed light on how different data properties affect the ability of population fidelity metrics to estimate utility and make recommendations on how to use these metrics for different scenarios and types of datasets.
We explore the use of various machine learning (ML) models for classifying lithologies utilizing data from X-ray fluorescence (XRF) and X-ray computed tomography (XCT). Typically, lithologies are identified over several meters, which restricts the use of ML models due to limited training data. To address this issue, we augment the original interval dataset, where lithologies are marked over extensive sections, into finer segments of 10cm, to produce a high resolution dataset with vastly increased sample size. Additionally, we examine the impact of adjacent lithologies on building a more generalized ML model. We also demonstrate that combining XRF and XCT data leads to an improved classification accuracy compared to using only XRF data, which is the common practice in current studies, or solely relying on XCT data.
The rapid growth of Large Language Models (LLMs), such as ChatGPT and Mistral, has raised concerns about their ability to generate inappropriate, toxic and ethically problematic content. This problem is further amplified by LLMs' tendency to reproduce the prejudices and stereotypes present in their training datasets, which include misinformation, hate speech and other unethical content. Traditional methods of automatic bias detection rely on static datasets that are unable to keep up with society's constantly changing prejudices, and so fail to capture the large diversity of biases, especially implicit associations related to demographic characteristics like gender, ethnicity, nationality, and so on. In addition, these approaches frequently use adversarial techniques that force models to generate harmful language. In response, this study proposes a novel qualitative protocol based on prompting techniques to uncover implicit bias in LLM-generated texts without explicitly asking for prejudicial content. Our protocol focuses on biases associated with gender, sexual orientation, nationality, ethnicity and religion, with the aim of raising awareness of the stereotypes perpetuated by LLMs. We include the Tree of Thoughts technique (ToT) in our protocol, enabling a systematic and strategic examination of internal biases. Through extensive prompting experiments, we demonstrate the effectiveness of the protocol in detecting and assessing various types of stereotypes, thus providing a generic and reproducible methodology. Our results provide important insights for the ethical evaluation of LLMs, which is essential in the current climate of rapid advancement and implementation of generative AI technologies across various industries.Warning: This paper contains explicit statements of offensiveor upsetting contents.
This study explores the effects of incorporating demonstrations as pre-training of an improved Deep Q-Network (DQN). Inspiration is taken from methods such as Deep Q-learning from Demonstrations (DQfD), but instead of retaining the demonstrations throughout the training, the performance and behavioral effects of the policy when using demonstrations solely as pre-training are studied. A comparative experiment is performed on two game environments, Gymnasium's Car Racing and Atari Space Invaders. While demonstration pre-training in Car Racing shows improved learning efficacy, as indicated by higher evaluation and training rewards, these improvements do not show in Space Invaders, where it instead under-performed. This divergence suggests that the nature of a game's reward structure influences the effectiveness of demonstration pre-training. Interestingly, despite less pronounced quantitative differences, qualitative observations suggested distinctive strategic behaviors, notably in target elimination patterns in Space Invaders. These retained behaviors seem to get forgotten during extended training. The results show that we need to investigate further how exploration functions affect the effectiveness of demonstration pre-training, how behaviors can be retained without explicitly making the agent mimic demonstrations, and how non-optimal demonstrations can be incorporated for more stable learning with demonstrations.
Shapley values have multiple desired and theoretically proven properties for explaining black-box model predictions. However, the exact computation of Shapley values can be computationally very expensive, precluding their use when timely explanations are required. FastSHAP is an approach for fast approximation of Shapley values using a trained neural network (the explainer). A novel approach, called FF-SHAP, is proposed, which incorporates three modifications to FastSHAP: i) the explainer is trained on ground-truth explanations rather than a weighted least squares characterization of the Shapley values, ii) cosine similarity is used as a loss function instead of mean-squared error, and iii) the actual prediction of the underlying model is given as input to the explainer. An empirical investigation is presented showing that FF-SHAP significantly outperforms FastSHAP with respect to fidelity, measured using both cosine similarity and Spearman's rank-order correlation. The investigation further shows that FF-SHAP even outperforms FastSHAP when using substantially smaller amounts of data to train the explainer, and more importantly, FF-SHAP still maintains the performance level of FastSHAP even when trained with as little as 15% of training data.
The opaque nature of machine learning systems has raised concerns about whether these systems can guarantee fairness. Furthermore, ensuring fair decision making requires the consideration of multiple perspectives on fairness.At the moment, there is no agreement on the definitions of fairness, achieving shared interpretations is difficult, and there is no unified formal language to describe them. Current definitions are implicit in the operationalization of systems, making their comparison difficult.In this paper, we propose a framework for specifying formal representations of fairness that allows instantiating, visualizing, and comparing different interpretations of fairness. Our framework provides a meta-model for comparative analysis. We present several examples that consider different definitions of fairness, as well as an open-source implementation that uses the object-oriented functional language Soda.
Value-based frameworks are widely used to guide the design of algorithms, yet their influence in mediating users’ perception and use of algorithm-driven technologies is vastly understudied. Moreover, there is a need to move research beyond a focus on human-algorithm interaction to account for how the values these frameworks promote – algorithmic values – become socialised outside the boundaries of the (human-algorithm) interaction and how they influence everyday practices that are not algorithmically mediated. This paper traces the entanglement of algorithmic values and everyday life by mapping how residents of the Salvadorian town of El Zonte perceive the top-down transition of the town into "Bitcoin Beach" through value-driven transformations to diverse aspects of their material culture and built environment. This approach advances empirical research on the impact of algorithms by acknowledging the myriad ways in which those who won’t or can’t (afford to) interact with algorithm-driven technologies are impacted by the value-based outcomes of their programming and provides novel insights for critically examining the role of algorithm-driven technologies in shaping sustainable futures.
Machine learning algorithms, particularly artificial neural networks, have shown promise in healthcare for disease classification, including diagnosing conditions like deep vein thrombosis. However, the performance of artificial neural networks in medical diagnosis heavily depends on their architecture and hyperparameter configuration, which presents virtually unlimited variations. This work employs evolutionary algorithms to optimize hyperparameters for three classic feed-forward artificial neural networks of pre-determined depths. The objective is to enhance the diagnostic accuracy of the classic neural networks in classifying deep vein thrombosis using electronic health records sourced from a Norwegian hospital. The work compares the predictive performance of conventional feed-forward artificial neural networks with standard tree-based ensemble methods previously successful in disease prediction on the same dataset. Results indicate that while classic neural networks perform comparably to tree-based methods, they do not surpass them in diagnosing thrombosis on this specific dataset. The efficacy of evolutionary algorithms in tuning hyperparameters is highlighted, emphasizing the importance of choosing the optimization technique to maximize machine learning models' diagnostic accuracy.
With 90% of the world's goods transported by sea vessels, it is crucial to investigate their safety. This is increasingly important as autonomy is being introduced into sea vessels, which transport goods and people. To study the safety of an autonomous ferry's collision avoidance system, we consider the Adaptive Stress Testing (AST) method in this work. AST uses machine learning, specifically reinforcement learning, along with a simulation of a system under test---in our case, an autonomous and electric ferry---and its environment. Whether that simulation is fully or partially observable has implications for the integration into existing engineering workflows. The reason is that the fully observable simulation induces a more complex interface than the partially observable simulation, meaning that the engineers designing and implementing AST need to acquire and comprehend more potentially complex domain knowledge. This paper presents maritime adaptive stress testing (MAST) methods, using the world's first autonomous, electric ferry used to transport people as a case study. Using MAST in multiple scenarios, we demonstrate that AST can be productively utilized in the maritime domain. The demonstration scenarios stress test a maritime collision avoidance system known as Single Path Velocity Planner (SP-VP). Additionally, we consider how MAST can be implemented to test using both fully observable (gray box) and partially observable (black box) simulators. Consequently, we introduce the Gray-Box MAST (G-MAST) and Black-Box MAST (B-MAST) architectures, respectively. In simulation experiments, both architectures successfully identify an almost equal number of failure events. We discuss lessons learned about MAST including the experiences with both the Gray-Box and Black-Box approaches.
In line with global sustainability goals, such as the Paris Agreement, accurate mapping, monitoring, and management of solar farms are critical for achieving net zero emissions by 2050. However, many solar installations remain undocumented, posing a challenge. This paper studies semantic segmentation using deep neural networks, including networks constructed using network architecture search (NAS), for solar farm detection. Semantic segmentation has evolved through technologies like Fully Convolutional Networks and U-Net, which have shown strong performance on satellite imagery. For NAS, Differentiable Architecture Search and its variants like Auto-DeepLab have become efficient ways to automate the creation of neural network architectures. This work compares models generated using Auto-DeepLab to Solis-seg, a Deep Neural Network optimized for detecting solar farms in satellite imagery. Solis-seg achieves a mean Intersection over Union (IoU) of 96.26% on a European Sentinel-2 dataset, with Auto-DeepLab models lagging slightly behind. Our results for Solis-seg also challenge the prevailing method of using transfer learning from classification tasks for semantic segmentation. Thus, this work contributes to both the field of earth observation machine learning and the global transition to renewable energy by studying an efficient, scalable approach to tracking solar installations. We believe that this paper offers valuable insights into applying advanced machine learning techniques to solar farm detection and can be useful for further research in earth observation and sustainability.
Detecting violations within fishing activity reports is crucial for ensuring the sustainable utilization of fish resources, and employing machine learning methods holds promise for uncovering hidden patterns within this complex dataset. Given that these violations are infrequent occurrences, as fishermen generally adhere to regulations, identifying them becomes akin to an anomaly outlier detection task. Since labeled data distinguishing between normal and anomalous instances is not available for catch reports from Norwegian waters, we have opted for more conventional approaches, such as clustering methods, to identify potential clusters and outliers. Moreover, the catch reports inherently exhibit randomness and noise due to environmental factors and potential errors made by fishermen during report registration which complicates the processes of scaling, clustering, and anomaly detection. Through experimentation with various scaling and clustering techniques, we have observed that many of these methods tend to group the data based on the species caught, exhibiting a high level of agreement in cluster formation, indicating the stability of the clusters. Anomaly detection methods, however, yield varying potential outliers as it is a more challenging task.
This study assesses two state-of-the-art (SOTA) pointcloud registration approaches on industrially challenging datasets, focusing on two specific cases. The first case involves the application of Lidar-based Simultaneous Localization and Mapping (SLAM) in a tunnel environment, while the second case revolves around aligning RGBD scans from intricately symmetrical cast-iron machine parts within the domain of small-scale industrial production. Our evaluation involves testing state-of-the-art pointcloud registration approaches both with and without fine-tuning, and comparing the results to a classical hand crafted feature extractors. Our experimental findings reveal that existing SOTA models exhibit limited generalization capability when confronted with the more challenging pointcloud data. Moreover, robust generalizable methods beyond training are currently unavailable, highlighting a notable gap in addressing challenges associated with industrial datasetsin pointcloud registration.
As the world of competitive sports increasingly embraces data-driven techniques, our research explores the potential of machine learning in distinguishing elite from semi-elite beach volleyball players. This study is motivated by the need to understand the subtle yet crucial differences in player movements that contribute to high-level performance in beach volleyball. Utilizing advanced machine learning techniques, we analyzed specific movement patterns of the motion of the torso during spikes, captured through vest-mounted accelerometers. Our approach offers novel insights into the nuanced dynamics of elite play, revealing that certain movement patterns are distinctly characteristic of higher skill levels. One of our key contributions is the ability to classify spiking movements at different skill levels with an accuracy rate as high as 87%. This current research provides a foundation of what separates elite players from their semi-elite counterparts.
Federated Learning (FL) is a decentralized learning paradigm, enabling parties to collaboratively train models while keeping their data confidential. Within autonomous driving, it brings the potential of reducing data storage costs, reducing bandwidth requirements, and to accelerate the learning. FL is, however, susceptible to poisoning attacks. In this paper, we introduce two novel poisoning attacks on FL tailored to regression tasks within autonomous driving: FLStealth and Off-Track Attack (OTA). FLStealth, an untargeted attack, aims at providing model updates that deteriorate the global model performance while appearing benign. OTA, on the other hand, is a targeted attack with the objective to change the global model's behavior when exposed to a certain trigger. We demonstrate the effectiveness of our attacks by conducting comprehensive experiments pertaining to the task of vehicle trajectory prediction. In particular, we show that, among five different untargeted attacks, FLStealth is the most successful at bypassing the considered defenses employed by the server. For OTA, we demonstrate the inability of common defense strategies to mitigate the attack, highlighting the critical need for new defensive mechanisms against targeted attacks within FL for autonomous driving.
Forecasting indoor temperatures is of paramount importance to achieve efficient control of HVAC systems. In this task, the limited data availability presents a challenge as most of the available data is acquired during standard operation where extreme scenarios and transitory regimes such as major temperature increases or decreases are de-facto excluded. Acquisition of such data requires significant energy consumption and a dedicated facility, hindering the quantity and diversity of available data. To acquire such data, we make use of such a facility referred to as the Test-cell. Cost related constraints however do not allow for continuous year-around acquisition.To address this, we investigate the efficacy of data augmentation techniques, particularly leveraging state-of-the-art AI-based methods for synthetic data generation. Inspired by practical and experimental motivations, we explore fusion strategies of real and synthetic data to improve forecasting models. This approach alleviates the need for continuously acquiring extensive time series data, especially in contexts involving repetitive heating and cooling cycles in buildings. Our evaluation methodology for synthetic data synthesis involves a dual-focused approach: firstly, we assess the performance of synthetic data generators independently, particularly focusing on SoTA AI-based methods; secondly, we measure the utility of incorporating synthetically augmented data in a subsequent downstream tasks (forecasting). In the forecasting tasks, we employ a simple model in two distinct scenarios: 1) we first examine an augmentation technique that combines real and synthetically generated data to expand the training dataset, 2) Second, we delve into utilizing synthetic data to tackle dataset imbalances. Our results highlight the potential of synthetic data augmentation in enhancing forecasting accuracy while mitigating training variance. Through empirical experiments, we show significant improvements achievable by integrating synthetic data, thereby paving the way for more robust forecasting models in low-data regime.
The widespread use of voice-assisted applications using artificial intelligence raises questions about the dynamics of trust and reliance on these systems. While users often rely on these applications for help, instances where users face unforeseen risks and heightened challenges have sparked conversations about the importance of fostering trustworthy artificial intelligence. In this paper, we argue that the prevailing narrative of trust and trustworthiness in relation to artificial intelligence, particularly voice assistants, is misconstrued and fundamentally misplaced. Drawing on insights from philosophy and artificial intelligence literature, we contend that artificial intelligence systems do not meet the criteria for participating in a relationship of trust with human users. Instead, a narrative of reliance is more appropriate. However, we investigate the matter further to explore why the trust/trustworthiness narrative persists, focusing on the unique social dynamics of interactions with voice assistants. We identify factors such as diverse modalities and complexity, social aspects of voice assistants, and issues of uncertainty, assertiveness, and transparency as contributors to the trust narrative. By disentangling these factors, we shed light on the complexities of human-computer interactions and offer insights into the implications for our relationship with artificial intelligence. We advocate for a nuanced understanding of trust and reliance in artificial intelligence systems and provide suggestions for addressing the challenges posed by the dominance of the trust/trustworthiness narrative.
