This work provides an overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis. After providing a brief overview of the key technologies behind the current generation of AI agents based on LLMs, we summarize the different approaches to evaluating these models in the context of materials science and in particular on materials synthesis, with a specific emphasis on scenarios in which the models are directly integrated with experimental tools. We discuss evaluation strategies spanning knowledge and reasoning benchmarks, tool-use benchmarks, and closed loop benchmarks involving the interaction with experimental systems or realistic virtual tools. We use atomic layer deposition (ALD) as a case study, emphasizing how existing approaches in the literature both build from general approaches used beyond materials science and can be generalized to other materials synthesis techniques. Finally, we provide a practical evaluation framework to evaluate LLMs in the context of materials synthesis

In this work, we explore the performance and behavior of AI agents based on reasoning large language models on atomic layer deposition (ALD) process optimization tasks. In these tasks, an agent has to iteratively explore a process configuration space to identify the optimal dose times for the precursor and the coreactant, generally without any prior knowledge about the process, including whether it is actually self-limited. The agent is meant to interact iteratively with an ALD reactor in a fully unsupervised way, receiving feedback on the results of the proposed experiments. We evaluate this agent using a simple model of an ALD tool that incorporates ALD processes with different self-limited surface reaction pathways as well as a nonself-limited component. Our results show that agents based on reasoning models like OpenAI’s o3 and GPT5 consistently succeeded at completing this optimization task, with a performance on par or superior to that of previous machine learning approaches. However, we observed significant run-to-run variability due to the nondeterministic nature of the model’s response and search strategy. In order to understand the logic followed by the reasoning model, we captured the reasoning language model’s open response detailing the reasoning process. An analysis of the responses showed that the logic of the model was sound and that its reasoning was based on the notions of self-limited process and saturation expected in the case of ALD. However, the agent can sometimes be misled by its own prior choices when exploring the optimization space, which contributes to the variability of the results of the optimization process.

In this work, we introduce the design of an atomic layer deposition (ALD) reactor augmented with an AI interface for autonomous materials synthesis. Our modular design encapsulates the particularities of the hardware behind a Python interface that communicates with the ALD control software via transmission control protocol. This interface is compatible with model context protocol interfaces used in agentic frameworks. We have integrated our tool with a simple AI agent that leverages a large language model to transform user-supplied queries into ALD processes that are then run in our reactor. Our approach uses a JavaScript object notation schema to encode ALD processes. Our experimental results show that the AI interface does not impose a significant overhead to our control software, at least within our fastest 10 ms scale. We also carried out a detailed evaluation of the agent performance using leading models in two classes of tasks: basic instruction and process discovery tasks, where the agent is presented with a target material and needs to identify the correct ALD process compatible with the reactor configuration. Despite the simplicity of our agent design, we observed that most of the advanced models excelled at the instruction tasks. However, only recent models, such as o1, o3, GPT-5, and Claude Opus 4, performed well in process discovery tasks. We also observed significant variability in the response for the hardest challenges. While the results obtained are promising, we identify areas where AI research could improve the performance of agents for ALD.

In this work, we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and, in particular, in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from the graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI’s GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1–5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, the specificity of the question, and the accuracy of the response as graded by the human experts. This emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.

Atomic layer deposition (ALD) is a promising technique to functionalize particle surfaces for energy applications including energy storage, catalysis, and decarbonization. In this work, we present a set of models of ALD particle coating to explore the transition from lab scale to manufacturing. Our models encompass the main particle coating manufacturing approaches including rotary bed, fluidized bed, and continuously vibrating reactors. These models provide key metrics, such as throughput and precursor utilization, required to evaluate the scalability of ALD manufacturing approaches and their feasibility in the context of energy applications. Our results show that designs that force the precursor to flow through fluidized particles transition faster to a transport-limited regime where throughput is maximized. They also exhibit higher precursor utilization. In the context of continuous processes, our models indicate that it is possible to achieve self-extinguishing processes with almost 100% precursor utilization. A comparison with past experimental results of ALD in fluidized bed reactors shows excellent qualitative and quantitative agreement.

In this work, we explore surrogate models to optimize plasma enhanced atomic layer deposition (PEALD) in high aspect ratio features. In plasma-based processes such as PEALD and atomic layer etching (ALE), surface recombination can dominate the reactivity of plasma species with the surface, which can lead to unfeasibly long exposure times to achieve full conformality inside nanostructures like high aspect ratio vias. Using a synthetic dataset based on simulations of PEALD, we train artificial neural networks to predict saturation times based on cross section thickness data obtained for partially coated conditions. The results obtained show that just two experiments in undersaturated conditions contain enough information to predict saturation times within 10% of the ground truth. A surrogate model trained to determine whether surface recombination dominates the plasma–surface interactions in a PEALD process achieves 99% accuracy. This demonstrates that machine learning can provide a new pathway to accelerate the optimization of PEALD processes in areas such as microelectronics. Our approach can be easily extended to ALE and more complex structures.

Fast process optimization is critical to help reduce the cost of development and adoption of new ALD processes. Two examples are energy technologies, where low cost manufacturing is key to ensure commercial viability, and microelectronics, where the cost of innovation is very high due to a combination of substrate complexity, sometimes requiring many prior processing steps, and ever stringent requirements in terms of conformality, roughness, and precision. In this presentation I will provide an overview of how machine learning and AI can help accelerate process optimization and materials discovery in atomic layer deposition. In particular, I will highlight three different approaches: the first one explores the use of surrogate models to connect experimental metrology data with optimal processing conditions. We have explored two different cases: optimizing a process within a reactor and optimizing process transfer to a different reactor. In both cases, we show that, for thermal ALD processes, the information contained in thickness profiles in undersaturated conditions is enough to help predict optimal dose times both within and across different reactors. We also extended this methodology to the case of plasma-assisted deposition processes. The second approach relies on the use of in-situ characterization techniques to design self-driving deposition tools that can automatically search and identify optimal process conditions. For this approach, we developed a two-step process where algorithms are tested first using simulations and digital twins of the reactors before being experimentally deployed. This methodology can lead to x100 faster process optimization compared to standard growth-vent-characterize optimization cycles. We are currently applying this approach to the growth of hybrid organic/inorganic materials using molecular layer deposition. Finally, I will focus on large language models and their potential to assist in ALD research, both in terms of their role as research assistants and in the design of experiments. For the former case, have we developed a benchmark focused on ALD that we have applied to GPT-4o model. For the latter, I will describe how you can integrate LLMs with experimental ALD tools. This research is based upon work supported by Laboratory Directed Research and Development (LDRD) funding from Argonne National Laboratory, provided by the Director, Office of Science, of the U.S. Department of Energy under Contract No. DE-AC02-06CH11357.

This work explores erbium-doped calcium molybdate (Er:CaMoO4) thin films grown on silicon and yttria stabilized zirconia (YSZ) substrates, as a potential solid state system for C-band (utilizing the ∼1.5 μm Er3+ 4f–4f transition) quantum emitters for quantum network applications. Through molecular beam epitaxial growth experiments and electron microscopy, X-ray diffraction, and reflection electron diffraction studies, we identify an incorporation limited deposition regime that enables a 1:1 Ca:Mo ratio in the growing film leading to single phase CaMoO4 formation that can be in situ doped with Er (typically 2–100 ppm). We further show that growth on silicon substrates is single phase but polycrystalline in morphology, while growth on YSZ substrates leads to high-quality epitaxial single crystalline CaMoO4 films. We perform preliminary optical and microwave characterization on the suspected Y1–Z1 transition of 2 ppm, 200 nm epitaxial Er:CaMoO4 annealed thin films and extract an optical inhomogeneous linewidth of 9.1(1) GHz, an optical excited state lifetime of 6.7(2) ms, a spectral diffusion-limited homogeneous linewidth of 6.7(4) MHz, and an EPR linewidth of 1.10(2) GHz.

The design of optimal sequences for multicomponent ALD processes can be a time-consuming procedure: transient effects can appear each time that one switches to a different precursor, making it hard to predict beforehand the desired composition. Consequently, researchers usually resort to trial runs to calibrate the film composition, and combination of precursors with well-behaved nucleation behaviors and fast transients are highly prized. While effective for processes involving two different precursors, this empirical approach makes it hard to extend ALD to explore more complex materials involving ternary or quaternary compounds. In this work we explore an alternative approach that uses machine learning integrated with in-situ techniques to optimize the composition of multicomponent films. Our approach relies on digital twins to train, optimize, and benchmark algorithms that build the sequence of ALD cycles in real time from the feedback of in-situ data. These models allow us to generate a wide range of processes with different types of nucleation behaviors, providing the ideal testbed for the development of robust algorithms. We then transfer these algorithms into our experimental reactors for the exploration and optimization of processes involving two and three different types of oxide materials. Our results show that it is possible to drive the optimization of binary and ternary ALD processes using solely the growth per cycle of the individual processes as input, with the algorithm being robust across different nucleation behaviors. The main limitation of the proposed approach is that, for techniques showing a net change in the material, such as quartz crystal microbalance, it is not possible to consider systems that exhibit a substantial amount of etching, such the ZnO/Al2O3 process involving diethyl zinc and trimethyaluminum precursors. This would not be a limitation for techniques providing direct compositional information, such as X-ray fluorescence, or those focusing on materials properties, such as in-situ spectroscopic ellipsometry. This research is based on the work supported by the Laboratory Directed Research and Development (LDRD) funding from the Argonne National Laboratory, provided by the Director, Office of Science, of the U.S. DOE under Contract No. DE-AC02-06CH11357.

Lifelong learning—an agent’s ability to learn throughout its lifetime—is a hallmark of biological learning systems and a central challenge for artificial intelligence (AI). The development of lifelong learning algorithms could lead to a range of novel AI applications, but this will also require the development of appropriate hardware accelerators, particularly if the models are to be deployed on edge platforms, which have strict size, weight and power constraints. Here we explore the design of lifelong learning AI accelerators that are intended for deployment in untethered environments. We identify key desirable capabilities for lifelong learning accelerators and highlight metrics to evaluate such accelerators. We then discuss current edge AI accelerators and explore the future design of lifelong learning accelerators, considering the role that different emerging technologies could play.

Atomic layer deposition (ALD) is a crucial technique in semiconductor miniaturization and high-precision applications. The quality of ALD processes directly affects the properties of the resulting thin films, leading to extensive evaluations for new ALD procedures. This study uses machine learning to quickly assess ALD process quality, with a focus on predicting the standard deviations of film thickness as an indicator of quality. Using a synthetic dataset simulating non-ideal ALD processes, we evaluated the performance of Random Forest Classifier (RFC), Support Vector Machines (SVMs), and K-Nearest Neighbor (KNN). We also introduced artificial neural network (ANN) and convolutional neural network (CNN) models for predicting standard deviations of film thickness from ALD trials. Our ANN and CNN models showed promising results, positioning them as reliable tools for predicting ALD process quality.

We present the Memristor-Spikelearn simulator (open-sourced), which is capable of incorporating detailed mem-ristor and circuit models in simulation to enable thorough study of synaptic plasticity in spiking neural networks under realistic device and circuit behaviors. Using this simulator, we demonstrate that: (1) a detailed device model is essential for simulating synaptic plasticity workloads, because results obtained using a simplified model can be misleading (e.g., it can overestimate test accuracy by up to 21.9%); (2) detailed simulation helps to determine the proper range of conductance values to represent weights, which is critical in order to achieve the desired accuracy -energy tradeoff (e.g., increasing the conductance values by $10\times$ can increase accuracy from 70% to 83% at the price of $20\times$ higher energy); and (3) detailed simulation also helps to determine an optimized circuit structure, which is another important design parameter that can yield different accuracy -energy tradeoffs.

Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed. This critical gap may be addressed through the development of "Lifelong Learning" systems that are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. Unfortunately, efforts to improve these capabilities are typically treated as distinct areas of research that are assessed independently, without regard to the impact of each separate capability on other aspects of the system. We instead propose a holistic approach, using a suite of metrics and an evaluation framework to assess Lifelong Learning in a principled way that is agnostic to specific domains or system techniques. Through five case studies, we show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems. We highlight how the proposed suite of metrics quantifies performance trade-offs present during Lifelong Learning system development - both the widely discussed Stability-Plasticity dilemma and the newly proposed relationship between Sample Efficient and Robust Learning. Further, we make recommendations for the formulation and use of metrics to guide the continuing development of Lifelong Learning systems and assess their progress in the future.

The ability to learn continuously from an incoming data stream without catastrophic forgetting is critical to designing intelligent systems. Many approaches to continual learning rely on stochastic gradient descent and its variants that employ global error updates, and hence need to adopt strategies such as memory buffers or replay to circumvent its stability, greed, and short-term memory limitations. To address this limitation, we have developed a biologically inspired lightweight neural network architecture that incorporates synaptic plasticity mechanisms and neuromodulation and hence learns through local error signals to enable online continual learning without stochastic gradient descent. Our approach leads to superior online continual learning performance on Split-MNIST, Split-CIFAR-10, and Split-CIFAR-100 datasets compared to other memory-constrained learning approaches and matches that of the state-of-the-art memory-intensive replay-based approaches. We further demonstrate the effectiveness of our approach by integrating key design concepts into other backpropagation-based continual learning algorithms, significantly improving their accuracy. Our results provide compelling evidence for the importance of incorporating biological principles into machine learning models and offer insights into how we can leverage them to design more efficient and robust systems for online continual learning.

In this work, we explore the application of deep neural networks to the optimization of atomic layer deposition (ALD) processes. In particular, we focus on a one-shot optimization problem, where we try to predict the optimal dose time that leads to saturation everywhere in the reactor based on thickness values measured at different points of an ALD reactor after a single trial growth. In order to tackle this problem, we introduce a dataset designed to train neural networks to predict saturation times based on these inputs for a cross-flow ALD reactor. We then explore the predictive ability of artificial neural networks of different depths and sizes using a separate testing dataset to evaluate their accuracies. The results obtained show that networks trained using stochastic gradient descent methods can accurately predict saturation times without requiring any additional information on the surface kinetics. This provides a viable approach to minimize the number of experiments required to optimize new ALD processes in a known reactor, and it highlights the way machine learning can be leveraged for thin film growth and manufacturing. While the datasets and training procedure depend on the reactor geometry, the trained neural networks provide a general surrogate model connecting thickness values and trial dose times with optimal saturation times that can be reused for different ALD processes within the same reactor.

We have developed a model for online continual or lifelong reinforcement learning (RL) inspired on the insect brain. Our model leverages the offline training of a feature extraction and a common general policy layer to enable the convergence of RL algorithms in online settings. Sharing a common policy layer across tasks leads to positive backward transfer, where the agent continuously improved in older tasks sharing the same underlying general policy. Biologically inspired restrictions to the agent's network are key for the convergence of RL algorithms. This provides a pathway towards efficient online RL in resource-constrained scenarios.

In this work we have extended AutoML inspired approaches to the exploration and optimization of neuromorphic architectures. Through the integration of a parallel asynchronous model-based search approach with a simulation framework to simulate spiking architectures, we are able to efficiently explore the configuration space of neuromorphic architectures and identify the subset of conditions leading to the highest performance in a targeted application. We have demonstrated this approach on an exemplar case of real time, on-chip learning application. Our results indicate that we can effectively use optimization approaches to optimize complex architectures, therefore providing a viable pathway towards application-driven codesign.