In reinforcement learning (RL), critics traditionally learn absolute state values, estimating how good a particular situation is in isolation. Adding any constant to $V(s)$ leaves action preferences unchanged. Thus only value differences are relevant for decision making. Motivated by this fact, we ask the question whether these differences can be learned directly. For this, we propose \emph{Relative Value Learning} (RV), a framework that considers antisymmetric value differences $\Delta(s_i, s_j) = V(s_i) - V(s_j)$. We define a new pairwise Bellman operator and prove it is a $\gamma$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step/$n$-step/$\lambda$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Besides rigorous theoretical contributions, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 games, ALE) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics. Source code will be made available.

In reinforcement learning, critics typically estimate absolute state values V(s), estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function Δ(s_i, s_j) = V(s_i) - V(s_j). We introduce a pairwise Bellman operator and prove it is a γ-contraction with a unique fixed point equal to the true value differences, derive well-posed 1-step, n-step and λ-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.

Clustered federated learning (FL) has emerged as an effective approach for handling heterogeneous data across clients by partitioning them into clusters with similar or identical data distributions. However, most existing methods, including the iterative federated clustering algorithm (IFCA), rely on a central server to coordinate model updates, typically requiring stable connectivity, synchronous communication rounds, and global aggregation of client models. These assumptions are difficult to satisfy in decentralized and heterogeneous environments, where clients may only have limited, local communication with a small subset of peers. As a result, such methods create a bottleneck and a single point of failure, limiting their applicability in realistic decentralized learning settings. This limitation is particularly severe in Internet of Things (IoT) settings, where large numbers of resource-constrained devices, intermittent or sparse connectivity, and dynamic participation make reliance on a central server impractical. In this work, we introduce the decentralized federated clustering algorithm (DFCA), a fully decentralized clustered FL algorithm that enables clients to collaboratively train cluster-specific models without central coordination. DFCA uses a sequential running average to aggregate models from neighbors as updates arrive, providing a communication-efficient alternative to batch aggregation while maintaining clustering performance. Our experiments on various datasets demonstrate that DFCA outperforms other decentralized algorithms and performs comparably to centralized IFCA, even under sparse connectivity, highlighting its robustness and practicality for dynamic real-world decentralized networks.

Political online participation in the form of discussing political issues and exchanging opinions among citizens is gaining importance with more and more formats being held digitally. To come to a decision, a thorough discussion and consideration of opinions and a civil exchange of arguments, which is defined as the act of deliberation, is desirable. The quality of discussions and participation processes in terms of their deliberativeness highly depends on the design of platforms and processes. To facilitate online communication for both participants and initiators, machine learning methods offer a lot of potential. In this work we want to showcase which issues occur in political online discussions and how machine learning can be used to counteract these issues and enhance deliberation. We conduct a literature review to (i) identify tasks that could potentially be solved by artificial intelligence (AI) algorithms to enhance individual aspects of deliberation in political online discussions, (ii) provide an overview on existing tools and platforms that are equipped with AI support and (iii) assess how well AI support currently works and where challenges remain.

Online spaces provide individuals with the opportunity to engage in discussions on important topics and make collective decisions, regardless of their geographic location or time zone. However, without adequate support and careful design, such discussions often suffer from a lack of structure and civility in the exchange of opinions. Artificial intelligence (AI) offers a promising avenue for helping both participants and organizers in managing large-scale online participation processes. This paper introduces an extension of adhocracy+, a large-scale open-source participation platform. Our extension features two AI-supported debate modules designed to improve discussion quality and foster participant interaction. In a large-scale user study we examined the effects and usability of both modules. We report our findings in this paper. The extended platform is available at https://github.com/mabehrendt/discuss2.0.

The [CLS] token in BERT is commonly used as a fixed-length representation for classification tasks, yet prior work has shown that both other tokens and intermediate layers encode valuable contextual information. In this work, we study lightweight extensions to BERT that refine the [CLS] representation by aggregating information across layers and tokens. Specifically, we explore three modifications: (i) max-pooling the [CLS] token across multiple layers, (ii) enabling the [CLS] token to attend over the entire final layer using an additional multi-head attention (MHA) layer, and (iii) combining max-pooling across the full sequence with MHA. Our approach, called MaxPoolBERT, enhances BERT's classification accuracy (especially on low-resource tasks) without requiring new pre-training or significantly increasing model size. Experiments on the GLUE benchmark show that MaxPoolBERT consistently achieves a better performance than the standard BERT base model on low resource tasks of the GLUE benchmark.

What are the essential components of world models? How far do we get with world models that are not employing RNNs, transformers, discrete representations, and image reconstructions? This paper introduces SGF, a Simple, Good, and Fast world model that uses self-supervised representation learning, captures short-time dependencies through frame and action stacking, and enhances robustness against model errors through data augmentation. We extensively discuss SGF's connections to established world models, evaluate the building blocks in ablation studies, and demonstrate good performance through quantitative comparisons on the Atari 100k benchmark.

BACKGROUND:Cardiac real-time MRI (RT-MRI) in combination with MR-compatible spirometry (MRcS) offers unique opportunities to study heart-lung interactions. In contrast to other techniques that monitor respiration during MRI, MRcS provides quantitative respiratory data. Though MRcS is well tolerated, shortening of the scanning time with MRcS would be desirable, especially in young and sick patients. PURPOSE:The aim of the study was to predict airflow and lung volume based on RT-MR images after a short learning phase of combined RT-MRI and MRcS to provide respiratory data for subsequent short axis stack-based volumetries. METHODS:Cardiac RT-MRI (1.5 T; short axis; 30 frames/s) was acquired during free breathing in combination with MRcS in adult healthy subjects (n = 10). MR images with MRcS were recorded during a learning phase to collect training data. The iterative Lucas-Kanade method was applied to estimate optical flow from the captured MR images. A ridge regression model was fitted to predict airflow and thus also the lung volume from the estimated optical flow. Hyperparameters were estimated using leave-one-out cross validation and the performance was assessed on a held-out test dataset. Different durations and compositions of the learning phase were investigated to develop the most efficient measurement protocol. Coefficient of determination (R2), relative mean squared error (rMSE), Bland-Altman analysis on absolute tidal volume difference (aTVD), and absolute maximal airflow difference (aMFD) were used to validate the predictions on held-out test data. RESULTS:MRI combined with MRcS can train a machine learning algorithm to provide excellent predictive quantitative respiratory volume and flow for the remaining study. The optimal trade-off between predictive power and time necessary for training was reached with a shortened cardiac volumetry protocol covering only about two breaths per slice and every second slice (airflow: mean R2: 0.984, mean rMSE: 0.015, Bias aMFD: -0.01 L/s with +0.084/-0.1 95% CI and volume: mean R2: 0.990, mean rMSE: 0.003, Bias aTVD: 4.27 mL with +33/-24 95% CI) at a total duration of 100 s. Shorter protocols or application of the algorithm to subsequent studies in the same subject or even in different subjects still provided useful qualitative data. CONCLUSION:Machine-learning-based prediction of respiratory flow and lung volume from cardiac RT-MR images after a short training phase with MRcS is feasible and can help to shorten the time with MRcS while providing accurate respiratory data during RT-MRI.

Object-centric learning is fundamental to human vision and crucial for models requiring complex reasoning. Traditional approaches rely on slot-based bottlenecks to learn object properties explicitly, while recent self-supervised vision models like DINO have shown emergent object understanding. However, DINO representations primarily capture global scene features, often confounding individual object attributes. We investigate the effectiveness of DINO representations and slot-based methods for multi-object instance retrieval. Our findings reveal that DINO representations excel at capturing global object attributes such as object shape and size, but struggle with object-level details like colour, whereas slot-based representations struggle at both global and object-level understanding. To address this, we propose a method that combines global and local features by augmenting DINO representations with object-centric latent vectors from a Variational Autoencoder trained on segmented image patches that are extracted from the DINO features. This approach improves multi-object instance retrieval performance, bridging the gap between global scene understanding and fine-grained object representation without requiring full model retraining.

Stance detection holds great potential to improve online political discussions through its deployment in discussion platforms for purposes such as content moderation, topic summarization or to facilitate more balanced discussions. Typically, transformer-based models are employed directly for stance detection, requiring vast amounts of data. However, the wide variety of debate topics in online political discussions makes data collection particularly challenging. LLMs have revived stance detection, but their online deployment in online political discussions faces challenges like inconsistent outputs, biases, and vulnerability to adversarial attacks. We show how LLM-generated synthetic data can improve stance detection for online political discussions by using reliable traditional stance detection models for online deployment, while leveraging the text generation capabilities of LLMs for synthetic data generation in a secure offline environment. To achieve this, (i) we generate synthetic data for specific debate questions by prompting a Mistral-7B model and show that fine-tuning with the generated synthetic data can substantially improve the performance of stance detection, while remaining interpretable and aligned with real world data. (ii) Using the synthetic data as a reference, we can improve performance even further by identifying the most informative samples in an unlabelled dataset, i.e., those samples which the stance detection model is most uncertain about and can benefit from the most. By fine-tuning with both synthetic data and the most informative samples, we surpass the performance of the baseline model that is fine-tuned on all true labels, while labelling considerably less data.

Learning meaningful representations is at the heart of many tasks in the field of modern machine learning. Recently, a lot of methods were introduced that allow learning of image representations without supervision. These representations can then be used in downstream tasks like classification or object detection. The quality of these representations is close to supervised learning, while no labeled images are needed. This survey paper provides a comprehensive review of these methods in a unified notation, points out similarities and differences of these methods, and proposes a taxonomy which sets these methods in relation to each other. Furthermore, our survey summarizes the most recent experimental results reported in the literature in form of a meta-study. Our survey is intended as a starting point for researchers and practitioners who want to dive into the field of representation learning.

In this paper we adopt a representation-centric perspective on exploration in reinforcement learning, viewing exploration fundamentally as a density estimation problem. We investigate the effectiveness of clustering representations for exploration in 3-D environments, based on the observation that the importance of pixel changes between transitions is less pronounced in 3-D environments compared to 2-D environments, where pixel changes between transitions are typically distinct and significant. We propose a method that performs episodic and global clustering on random representations and on pre-trained DINO representations to count states, i.e, estimate pseudo-counts. Surprisingly, even random features can be clustered effectively to count states in 3-D environments, however when these become visually more complex, pre-trained DINO representations are more effective thanks to the pre-trained inductive biases in the representations. Overall, this presents a pathway for integrating pre-trained biases into exploration. We evaluate our approach on the VizDoom and Habitat environments, demonstrating that our method surpasses other well-known exploration methods in these settings.

We present a Deep Learning approach to predict 3D folding structures of RNAs from their nucleic acid sequence. Our approach combines an autoregressive Deep Generative Model, Monte Carlo Tree Search, and a score model to find and rank the most likely folding structures for a given RNA sequence. We show that RNA de novo structure prediction by deep learning is possible at atom resolution, despite the low number of experimentally measured structures that can be used for training. We confirm the predictive power of our approach by achieving competitive results in a retrospective evaluation of the RNA-Puzzles prediction challenges, without using structural contact information from multiple sequence alignments or additional data from chemical probing experiments. Blind predictions for recent RNA-Puzzle challenges under the name "Dfold" further support the competitive performance of our approach.

Stance detection is an important task for many applications that analyse or support online political discussions. Common approaches include fine-tuning transformer based models. However, these models require a large amount of labelled data, which might not be available. In this work, we present two different ways to leverage LLM-generated synthetic data to train and improve stance detection agents for online political discussions: first, we show that augmenting a small fine-tuning dataset with synthetic data can improve the performance of the stance detection model. Second, we propose a new active learning method called SQBC based on the "Query-by-Comittee" approach. The key idea is to use LLM-generated synthetic data as an oracle to identify the most informative unlabelled samples, that are selected for manual labelling. Comprehensive experiments show that both ideas can improve the stance detection performance. Curiously, we observed that fine-tuning on actively selected samples can exceed the performance of using the full dataset.

Measuring the quality of contributions in political online discussions is crucial in deliberation research and computer science. Research has identified various indicators to assess online discussion quality, and with deep learning advancements, automating these measures has become feasible. While some studies focus on analyzing specific quality indicators, a comprehensive quality score incorporating various deliberative aspects is often preferred. In this work, we introduce AQuA, an additive score that calculates a unified deliberative quality score from multiple indices for each discussion post. Unlike other singular scores, AQuA preserves information on the deliberative aspects present in comments, enhancing model transparency. We develop adapter models for 20 deliberative indices, and calculate correlation coefficients between experts' annotations and the perceived deliberativeness by non-experts to weigh the individual indices into a single deliberative score. We demonstrate that the AQuA score can be computed easily from pre-trained adapters and aligns well with annotations on other datasets that have not be seen during training. The analysis of experts' vs. non-experts' annotations confirms theoretical findings in the social science literature.

AbstractRecent advances in sensing technologies, particularly Distributed Acoustic Sensing (DAS), have significantly improved the collection and analysis of seismological data. DAS is a powerful method for detecting vibrations from various sources, including earthquakes. However, the vast amount of data produced by DAS requires sophisticated analytical methods to differentiate between signals of interest and noise, such as traffic signals. We introduce an innovative approach by extending the Noise2Self framework to effectively remove unwanted, structured coherent noise from DAS data. By creating masks based on the characteristics of traffic signals, we isolate and preserve earthquake signals while maintaining the denoising performance of the original Noise2Self approach, which reduces noise without requiring clean reference data. To evaluate our method, we used synthetic data generated from seismic recordings of closely spaced seismometers and then applied our approach to data from a DAS array crossing the Alpine Fault near Haast, New Zealand. Our results demonstrate that our model successfully removes traffic noise and other non‐coherent noise while preserving seismic signals, leading to improvements in both Signal‐to‐Noise Ratio (SNR) and waveform coherence. Evaluations on real‐world DAS data further confirm the robustness of our method, positioning it as a valuable tool for analyzing large‐scale DAS data sets in various geoscientific contexts. This approach, which we refer to as “CoherentNoise2Self” to emphasize the extension of Noise2Self to coherent noise, advances the capabilities for near real‐time monitoring and analysis of seismic DAS data.

Phase retrieval is the problem of reconstructing images from magnitude-only measurements. In many real-world applications the problem is underdetermined. When training data is available, generative models allow optimization in a lower-dimensional latent space, hereby constraining the solution set to those images that can be synthesized by the generative model. However, not all possible solutions are within the range of the generator. Instead, they are represented with some error. To reduce this representation error in the context of phase retrieval, we first leverage a novel variation of intermediate layer optimization (ILO) to extend the range of the generator while still producing images consistent with the training data. Second, we introduce new initialization schemes that further improve the quality of the reconstruction. With extensive experiments on the Fourier phase retrieval problem and thorough ablation studies, we can show the benefits of our modified ILO and the new initialization schemes. Additionally, we analyze the performance of our approach on the Gaussian phase retrieval problem.

In this paper, we present our approach for the Helsinki Deblur Challenge (HDC2021). The task of this challenge is to deblur images of characters without knowing the point spread function (PSF). The organizers provided a dataset of pairs of sharp and blurred images. Our method consists of three steps: First, we estimate a warping transformation of the images to align the sharp images with the blurred ones. Next, we estimate the PSF using a quasi-Newton method. The estimated PSF allows to generate additional pairs of sharp and blurred images. Finally, we train a deep convolutional neural network to reconstruct the sharp images from the blurred images. Our method is able to successfully reconstruct images from the first 10 stages of the HDC 2021 data. Our code is available at https://github.com/hhu-machine-learning/hdc2021-psfnn.

Many production processes are characterized by numerous and complex cause-and-effect relationships. Since they are only partially known they pose a challenge to effective process control. In this work we present how Structural Equation Models can be used for deriving cause-and-effect relationships from the combination of prior knowledge and process data in the manufacturing domain. Compared to existing applications, we do not assume linear relationships leading to more informative results.

Social media platforms are often accused of disproportionally exposing their users to like-minded opinions, thereby fueling political polarization. However, empirical evidence of this causal relationship is inconsistent at best. One reason could be that many previous studies were unable to separate the effects caused by individual exposure to like-minded content from the effects caused by the algorithms themselves. This study presents results from two quasi-experiments in which participants were exposed either to algorithmically selected or randomly selected arguments that were either in line or in contrast with their attitudes on two different topics. The results reveal that exposure to like-minded arguments increased participants’ attitude polarization and affective polarization more intensely than exposure to opposing arguments. Yet, contrary to popular expectations, these effects were not amplified by algorithmic selection. Still, for one topic, exposure to algorithmically selected arguments led to slightly stronger attitude polarization than randomly selected arguments.