We introduce Emergent Trust Learning (ETL), a lightweight, trust-based control algorithm that can be plugged into existing AI agents. It enables these to reach cooperation in competitive game environments under shared resources. Each agent maintains a compact internal trust state, which modulates memory, exploration, and action selection. ETL requires only individual rewards and local observations and incurs negligible computational and communication overhead. We evaluate ETL in three environments: In a grid-based resource world, trust-based agents reduce conflicts and prevent long-term resource depletion while achieving competitive individual returns. In a hierarchical Tower environment with strong social dilemmas and randomised floor assignments, ETL sustains high survival rates and recovers cooperation even after extended phases of enforced greed. In the Iterated Prisoner's Dilemma, the algorithm generalises to a strategic meta-game, maintaining cooperation with reciprocal opponents while avoiding long-term exploitation by defectors. Code will be released upon publication.

Computer-aided synthesis planning aims to identify viable synthetic routes from a target compound to readily available building blocks by iteratively decomposing molecules into smaller precursors. Self-play search algorithms, trained with simulated experience, reach state-of-the-art performance. However, these methods typically plan in the molecular rather than the reaction space, leading to redundant or near-duplicate reaction outcomes in the search tree. In this work, we introduce a reaction-centric planning approach that measures the novelty of proposed reactions to analyze the synthesis planning search problem, constraining the search problem to genuinely unexplored disconnection ideas, i.e., unique ways of decomposing molecules using reactions. Our results show that the overall synthesis planning search space is much smaller than expected due to the absence of diverse disconnection ideas within the underlying template-based retrosynthesis model. Surprisingly, we also find that, under a reasonable time budget of less than an hour, online search algorithms outperform state-of-the-art self-play methods and are more robust to environmental changes, such as minor modifications to the available purchasable building blocks. Finally, we show that the diversity of the synthesis route solution space saturates when combining the results of different search algorithms, highlighting the importance of the single-step model in providing novel and chemically valid disconnections.

In real-world applications, users often favor structurally diverse design choices over one high-quality solution. It is therefore important to consider more solutions that decision makers can compare and further explore based on additional criteria. Alongside the existing approaches of evolutionary diversity optimization, quality diversity, and multimodal optimization, this paper presents a fresh perspective on this challenge by considering the problem of identifying a fixed number of solutions with a pairwise distance above a specified threshold while maximizing their average quality. We obtain first insight into these objectives by performing a subset selection on the search trajectories of different well-established search heuristics, whether they have been specifically designed with diversity in mind or not. We emphasize that the main goal of our work is not to present a new algorithm but to understand the capability of off-the-shelf algorithms to quantify the trade-off between the minimum pairwise distance within batches of solutions and their average quality. We also analyze how this trade-off depends on the properties of the underlying optimization problem. A possibly surprising outcome of our empirical study is the observation that naive uniform random sampling establishes a very strong baseline for our problem, hardly ever outperformed by the search trajectories of the considered heuristics. We interpret these results as a motivation to develop algorithms tailored to produce diverse solutions of high average quality.

This paper develops and analyses a novel quantum combinatorial game: quantum checkers (named Cheqqers). The concepts of superposition, entanglement, measurements and interference from quantum mechanics are integrated into the game of checkers by adding new types of legal moves. The addition of these new rules is done gradually by introducing several levels of 'quantumness'. Quantum checkers provides a framework for interpolating between a known and solved classical game and a more complex quantum game, and serves as 1) a benchmark for AI players learning to play quantum games and 2) an interesting game for human players that allows them to build intuition for quantum phenomena. We provide the initial analysis on the complexity of this game using random agents and a Monte Carlo tree search agent.

Rather than obtaining a single good solution for a given optimization problem, users often seek alternative design choices, because the best-found solution may perform poorly with respect to additional objectives or constraints that are difficult to capture into the modeling process. Aiming for batches of diverse solutions of high quality is often desirable, as it provides flexibility to accommodate post-hoc user preferences. At the same time, it is crucial that the quality of the best solution found is not compromised. One particular problem setting balancing high quality and diversity is fixing the required minimum distance between solutions while simultaneously obtaining the best possible fitness. Recent work by Santoni et al. [arXiv 2024] revealed that this setting is not well addressed by state-of-the-art algorithms, performing in par or worse than pure random sampling. Driven by this important limitation, we propose a new approach, where parallel runs of the covariance matrix adaptation evolution strategy (CMA-ES) inherit tabu regions in a cascading fashion. We empirically demonstrate that our CMA-ES-Diversity Search (CMA-ES-DS) algorithm generates trajectories that allow to extract high-quality solution batches that respect a given minimum distance requirement, clearly outperforming those obtained from of-the-shelf random sampling, multi-modal optimization algorithms, and standard CMA-ES.

Applications of machine learning in chemistry are often limited by the scarcity and expense of labeled data, restricting traditional supervised methods. In this work, we introduce a framework for molecular reasoning using general-purpose Large Language Models (LLMs) that operates without requiring labeled training data. Our method anchors chain-of-thought reasoning to the molecular structure by using unique atomic identifiers. First, the LLM performs a one-shot task to identify relevant fragments and their associated chemical labels or transformation classes. In an optional second step, this position-aware information is used in a few-shot task with provided class examples to predict the chemical transformation. We apply our framework to single-step retrosynthesis, a task where LLMs have previously underperformed. Across academic benchmarks and expert-validated drug discovery molecules, our work enables LLMs to achieve high success rates in identifying chemically plausible reaction sites (≥90%), named reaction classes (≥40%), and final reactants (≥74%). Beyond solving complex chemical tasks, our work also provides a method to generate theoretically grounded synthetic datasets by mapping chemical knowledge onto the molecular structure and thereby addressing data scarcity.

Background: There is great interest in agentic LLMs, large language models that act as agents. Objectives: We review the growing body of work in this area and provide a research agenda. Methods: Agentic LLMs are LLMs that (1) reason, (2) act, and (3) interact. We organize the literature according to these three categories. Results: The research in the first category focuses on reasoning, reflection, and retrieval, aiming to improve decision making; the second category focuses on action models, robots, and tools, aiming for agents that act as useful assistants; the third category focuses on multi-agent systems, aiming for collaborative task solving and simulating interaction to study emergent social behavior. We find that works mutually benefit from results in other categories: retrieval enables tool use, reflection improves multi-agent collaboration, and reasoning benefits all categories. Conclusions: We discuss applications of agentic LLMs and provide an agenda for further research. Important applications are in medical diagnosis, logistics and financial market analysis. Meanwhile, self-reflective agents playing roles and interacting with one another augment the process of scientific research itself. Further, agentic LLMs provide a solution for the problem of LLMs running out of training data: inference-time behavior generates new training states, such that LLMs can keep learning without needing ever larger datasets. We note that there is risk associated with LLM assistants taking action in the real world-safety, liability and security are open problems-while agentic LLMs are also likely to benefit society.

Since the release of GPT in 2022, generative artificial intelligence (AI) has quickly become ubiquitous. In educational institutions, the impact of generative AI has almost monopolised internal debate. Most of us have probably been involved in discussions about how to tackle the growing reliance of students on large language models (LLM). Research on the impact of this technology in education is catching up. In the case of computer science education, experimental projects in the field report contradictory results. This paper is composed of two parts: it first explores existing literature and compares studies to shed light on the impact of generative AI specifically for programming education. In doing so, we highlight the risks arising from uncontrolled generative AI. However, we also argue in favour of controlled generative AI as a tutoring tool. This leads us to the second part; we involve experts in a preliminary experiment to investigate how prompting can be used to align AI models with specific educational styles. Initial results suggest that detailed role-playing instructions, built upon existing pedagogical research, lead to more effective feedback.

Learning diverse skills without hand-crafted reward functions could accelerate reinforcement learning in downstream tasks. However, existing skill discovery methods focus solely on maximizing the diversity of skills without considering human preferences, which leads to undesirable behaviors and possibly dangerous skills. For instance, a cheetah robot trained using previous methods learns to roll in all directions to maximize skill diversity, whereas we would prefer it to run without flipping or entering hazardous areas. In this work, we propose a Foundation model Guided (FoG) skill discovery method, which incorporates human intentions into skill discovery through foundation models. Specifically, FoG extracts a score function from foundation models to evaluate states based on human intentions, assigning higher values to desirable states and lower to undesirable ones. These scores are then used to re-weight the rewards of skill discovery algorithms. By optimizing the re-weighted skill discovery rewards, FoG successfully learns to eliminate undesirable behaviors, such as flipping or rolling, and to avoid hazardous areas in both state-based and pixel-based tasks. Interestingly, we show that FoG can discover skills involving behaviors that are difficult to define. Interactive visualisations are available from https://sites.google.com/view/submission-fog.

We introduce a multi-objective search algorithm for retrosynthesis planning, based on a Monte Carlo Tree search formalism. The multi-objective search allows for combining diverse set of objectives without considering their scale or weighting factors. To benchmark this novel algorithm, we employ four objectives in a total of eight retrosynthesis experiments on a PaRoutes benchmark set. The objectives range from simple ones based on starting material and step count to complex ones based on synthesis complexity and route similarity. We show that with the careful employment of complex objectives, the multi-objective algorithm can outperform the single-objective search and provides amore diverse set of solutions. However, for many target compounds, the single- and multi-objective settings are equivalent. Nevertheless, our algorithm provides a framework for incorporating novel objectives for specific applications in synthesis planning.

We present a procedural content generation (PCG) pipeline for Celeste, a complex 2D platformer with horizontal and vertical movement and with limited prior AI framework development. Our approach utilizes a Markov Chain-based model to capture the game's unique structural and gameplay elements, generating playable levels that adhere to Celeste's design principles. We implemented post-processing steps to enhance playability and strategically place game elements. Our evaluation metrics focused on playability and interestingness, with results that indicate success in replicating the desired gameplay experience for beginner players. The evaluation involved 12 players of different skill levels, providing insight into the effectiveness of our generated content. Although some limitations were observed, such as occasional lack of creativity and difficulty in controlling challenge levels, our pipeline demonstrates promise as a foundation for a Celeste AI framework. This study contributes to the broader field of PCG for complex platformers and opens avenues for level generation in which agent-based evaluation is not feasible.

Unlike multiobjective optimization (MOO), multimodal multiobjective optimization (MMMOO) should approximate the entire Pareto set, even if a portion of it maps onto the entire Pareto front. This study introduces a novel evolution strategy with adaptive fitness sharing (AFS) for MMMOO. The method, called, AFS-MMMO-ES, calculates an overall fitness for each solution in the selection pool based on its rank-wise hypervolume contribution, Pareto rank, and niche count in the decision space. Since the optimal niche radius is problem-dependent, this study introduces a novel strategy for on-the-fly adaptation of the niche radius. Simulations on meticulously designed test problems are performed to confirm the efficacy and reliability of this strategy in learning the optimal niche radius, as well as its significant impact on enhancing robustness and performance. Furthermore, AFS-MMMO-ES can easily reflect the relative importance of decision space diversity based on the decision-maker’s preference, a practically important feature that has been overlooked in this research field. Finally, the performance of AFS-MMMO-ES is assessed and compared with several successful MMMOO methods on a widely accepted test suite for MMMOO. Comparisons of numerical results reveal the robustness and superiority of AFS-MMMO-ES over its competitors.

We present a Deep Reinforcement Learning (DRL) agent that successfully completes the first several hours of Pokemon Red, a classic Game Boy JRPG that exposes significant challenges as a testbed for agents, including multitasking, long horizons of tens of thousands of steps, hard exploration, and a vast array of potential policies. Our agent completes an initial segment of the game, up to Cerulean City, a location requiring progression through two cities, battle-filled passages, a maze-like cave, and defeating the first gym leader. Our experiments include various ablations that reveal vulnerabilities in reward shaping. We argue that long-form games like Pokemon hold strong potential for future research, presenting long-term coherent reasoning challenges absent from simpler arcade games. Our environment wrapper, training algorithm, human replay data, and pretrained agent are available at (REDACTED FOR REVIEW).

Molecules generated by Computer-Aided Drug Design often lack synthesizability to be valuable because Computer-Aided Synthesis Planning (CASP) and CASP-based approximated synthesizability scores have rarely been used as generation objectives, despite facilitating the in-silico generation of synthesizable molecules. Published scores approximate a general notion of CASP-based synthesizability with nearly unlimited building block resources. However, this approach is disconnected from the reality of small laboratory drug design, where building block resources are limited, making a notion of in-house synthesizability that uses already available resources highly desirable. In this work, we show a successful de novo drug design workflow generating active and in-house synthesizable ligands of monoglyceride lipase (MGLL). We demonstrate the successful transfer of CASP from 17.4 million commercial building blocks to a small laboratory setting of roughly 6,000 building blocks with only a decrease of -12% in CASP success. Moreover, we present a rapidly retrainable in-house synthesizability score, successfully capturing our in-house synthesizability without relying on external building block resources. We show that including our in-house synthesizability score in a multi-objective de novo drug design workflow, alongside a simple QSAR model, provides thousands of potentially active and easily in-house synthesizable molecules. Further, we highlight differences between general and in-house synthesizability scores and demonstrate potential problems with the out-of-distribution predictive performance of synthesizability scores on generated molecules. Finally, we experimentally evaluate the synthesis and biochemical activity of three de novo candidates using their CASP-suggested synthesis routes using only in-house building blocks. We find one candidate with evident activity, suggesting potential new ligand ideas for MGLL inhibitors while showcasing the usefulness of our in-house synthesizability score.

Memory Gym presents a suite of 2D partially observable environments, namely Mortar Mayhem, Mystery Path, and Searing Spotlights, designed to benchmark memory capabilities in decision-making agents. These environments, originally with finite tasks, are expanded into innovative, endless formats, mirroring the escalating challenges of cumulative memory games such as “I packed my bag”. This progression in task design shifts the focus from merely assessing sample efficiency to also probing the levels of memory effectiveness in dynamic, prolonged scenarios. To address the gap in available memory-based Deep Reinforcement Learning baselines, we introduce an implementation that integrates Transformer-XL (TrXL) with Proximal Policy Optimization. This approach utilizes TrXL as a form of episodic memory, employing a sliding window technique. Our comparative study between the Gated Recurrent Unit (GRU) and TrXL reveals varied performances across different settings. TrXL, on the finite environments, demonstrates superior sample efficiency in Mystery Path and outperforms in Mortar Mayhem. However, GRU is more efficient on Searing Spotlights. Most notably, in all endless tasks, GRU makes a remarkable resurgence, consistently outperforming TrXL by significant margins. Website and Source Code: https://github.com/MarcoMeter/endless-memory-gym/

Advances in deep learning have expanded the applications of virtual screening for drug-like compounds. More recently generative models have emerged as sources of inspiration for chemists. We introduce a multi-target model, PCMol, that leverages the latent embeddings derived from AlphaFold as a means of conditioning the de novo generative model on target proteins. It is known that the addition of protein descriptors is an effective strategy to extend the applicability domain and prediction capability of quantitative structure-activity relation (QSAR) models, a strategy we refer to as proteochemometrics (PCM). Similarly, the use of AlphaFold latent embeddings within a generative model for small molecules allows it to leverage structural relationships between proteins. This opens up new possibilities such as interpolation within the chemical space of known highly active compounds and extrapolation on the target side based on their similarities to other proteins, which is especially relevant for understudied or novel targets. Our results indicate that PCMol can generate diverse, potentially active molecules for a wide array of proteins, including those with sparse ligand bioactivity data. We also benchmark against existing target-conditioned trans-former models to illustrate the validity of using AlphaFold protein representations to steer the molecular generation process and increase the generalization capabilities to unseen targets. Additionally, we demonstrate the important role of data augmentation in bolstering the performance of generative models in low-data regimes. The open-source package along with a dataset of AlphaFold protein embeddings is available at https://github.com/CDDLeiden/PCMol.

This work presents the experimental setup for the GECCO 2024 competition on benchmarking niching methods for multimodal optimization. It employs 16 recently developed scalable composite problems to evaluate different aspects of multimodal optimization methods. These test problems are available in Python and MATLAB.

This research study explores the emotional responses evoked during game play and interactions with non-player characters (NPCs) in a mystery-solving game. Structured around three phases-introduction, collaboration, and feedback-, the game uses large language models (LLMs) to simulate varying emotional states in NPCs, from neutrality to expressions of anger, joy, and more. Players’ emotional responses to the game play and interaction with the NPCs are captured through the dialogue they input. Language models are used to extract emotion scores from these in-game conversations. This study aims to enhance our understanding of emotional dynamics within gaming environments, in order to further inform the design of emotionally engaging experiences. Additionally, it underscores the potential of language models for scientific inquiries in human-computer interaction.

Procedural Content Generation algorithms that make use of Machine Learning have garnered significant attention from the general public due to their ability to generate text, images, or video content that is often indistinguishable from human-crafted artworks. These achievements typically necessitate a substantial quantity of data, which may be scarce in specialized domains such as game-level design. One machine learning technique, Reinforcement Learning (RL), can be employed to learn from trial and error to address this scarcity. However, the RL training process requires a substantial time investment and may prove unsuccessful in the case of sparse reward structures. This paper empirically demonstrates the efficacy of curriculum learning as a viable solution to address scalability issues inherent to learning more complex tasks. Instead of trying to learn the whole space from scratch, we employ transfer learning on a curriculum from small to larger levels. We empirically validated this in a 3D vector-based environment, where the objective is to generate free-form tracks that facilitate a rider to move between designated points in the same vain as the flash game hit “Linerider”.