This research focuses on enhancing reinforcement learning (RL) algorithms by integrating penalty functions to guide agents in avoiding unwanted actions while optimizing rewards. The goal is to improve the learning process by ensuring that agents learn not only suitable actions but also which actions to avoid. Additionally, we reintroduce a bidirectional learning approach that enables agents to learn from both initial and terminal states, thereby improving speed and robustness in complex environments. Our proposed Penalty-Based Bidirectional methodology is tested against Mani skill benchmark environments, demonstrating an optimality improvement of success rate of approximately 4 that this integrated strategy enhances policy learning, adaptability, and overall performance in challenging scenarios

Translating natural language (NL) into a formal language such as temporal logic (TL) is integral for human communication with robots and autonomous systems. State-of-the-art approaches decompose the task into a grounding of atomic propositions (APs) phase and a translation phase. However, existing methods struggle with accurate grounding, the existence of co-references, and learning from limited data. In this paper, we propose a framework for NL to TL translation called Grammar Forced Translation (GraFT). The framework is based on the observation that previous work solves both the grounding and translation steps by letting a language model iteratively predict tokens from its full vocabulary. In contrast, GraFT reduces the complexity of both tasks by restricting the set of valid output tokens from the full vocabulary to only a handful in each step. The solution space reduction is obtained by exploiting the unique properties of each problem. We also provide a theoretical justification for why the solution space reduction leads to more efficient learning. We evaluate the effectiveness of GraFT using the CW, GLTL, and Navi benchmarks. Compared with state-of-the-art translation approaches, it can be observed that GraFT improves the end-to-end translation accuracy by 5.49% and out-of-domain translation accuracy by 14.06% on average.

Automated Program Repair tools are developed for generating feedback and suggesting a repair method for erroneous code. State of the art (SOTA) code repair methods rely on data-driven approaches and often fail to deliver solution for complicated programming questions. To interpret the natural language of unprecedented programming problems, using Large Language Models (LLMs) for code-feedback generation is crucial. LLMs generate more comprehensible feedback than compiler-generated error messages, and Reinforcement Learning with Human Feedback (RLHF) further enhances quality by integrating human-in-the-loop which helps novice students to lean programming from scratch interactively. We are applying RLHF fine-tuning technique for an expected Socratic response such as a question with hint to solve the programming issue. We are proposing code feedback generation tool by fine-tuning LLM with RLHF, Automated Code Evaluation with RLHF (ACE-RLHF), combining two open-source LLM models with two different SOTA optimization techniques. The quality of feedback is evaluated on two benchmark datasets containing basic and competition-level programming questions where the later is proposed by us. We achieved 2-5 using Llama-3-7B-Proximal-policy optimization in automated evaluation and similar or slightly higher accuracy compared to reward model-free RL with AI Feedback (RLAIF). We achieved almost 40 optimization while performing manual evaluation.

In-memory processing offers a promising solution for enhancing the performance of data-intensive applications. While analog in-memory computing demonstrates remarkable efficiency, its limited precision is suitable only for approximate computing tasks. In contrast, digital in-memory computing delivers the deterministic precision necessary to accelerate high-assurance applications. Current digital in-memory computing methods typically involve manually breaking down arithmetic operations into in-memory compute kernels. In contrast, traditional digital circuits are synthesized through intricate and automated design workflows. In this article, we introduce a logic synthesis framework called LOGIC, which facilitates the translation of high-level applications into digital in-memory compute kernels that can be executed using non-volatile memory. We propose techniques for decomposing element-wise arithmetic operations into in-memory kernels while minimizing the number of in-memory operations. Additionally, we optimize the sequence of in-memory operations to reduce non-volatile memory utilization. To address the NP-hard execution sequencing optimization problem, we have developed two look-ahead algorithms that offer practical solutions. Additionally, we leverage data layout reorganization to efficiently accelerate applications that heavily rely on sparse matrix-vector multiplication operations. Our experimental evaluations demonstrate that our proposed synthesis approach improves the area and latency of fixed-point multiplication by 84% and 20% compared to the state-of-the-art, respectively. Moreover, when applied to scientific computing applications sourced from the SuiteSparse Matrix Collection, our design achieves remarkable improvements in area, latency, and energy efficiency by factors of 4.8×, 2.6×, and 11×, respectively.

This paper presents a novel deep learning framework for robotic arm manipulation that integrates multimodal inputs using a late-fusion strategy. Unlike traditional end-to-end or reinforcement learning approaches, our method processes image sequences with pre-trained models and robot state data with machine learning algorithms, fusing their outputs to predict continuous action values for control. Evaluated on BridgeData V2 and Kuka datasets, the best configuration (VGG16 + Random Forest) achieved MSEs of 0.0021 and 0.0028, respectively, demonstrating strong predictive performance and robustness. The framework supports modularity, interpretability, and real-time decision-making, aligning with the goals of adaptive, human-in-the-loop cyber-physical systems.

We propose MORAL (a multimodal reinforcement learning framework for decision making in autonomous laboratories) that enhances sequential decision-making in autonomous robotic laboratories through the integration of visual and textual inputs. Using the BridgeData V2 dataset, we generate fine-tuned image captions with a pretrained BLIP-2 vision-language model and combine them with visual features through an early fusion strategy. The fused representations are processed using Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) agents. Experimental results demonstrate that multimodal agents achieve a 20% improvement in task completion rates and significantly outperform visual-only and textual-only baselines after sufficient training. Compared to transformer-based and recurrent multimodal RL models, our approach achieves superior performance in cumulative reward and caption quality metrics (BLEU, METEOR, ROUGE-L). These results highlight the impact of semantically aligned language cues in enhancing agent learning efficiency and generalization. The proposed framework contributes to the advancement of multimodal reinforcement learning and embodied AI systems in dynamic, real-world environments.

Vision-language models have demonstrated extraordinary zero-shot image classification capabilities. Out-of-distribution (OOD) detection is the problem of determining if a model is operating within its knowledge limits. While distance-based detection algorithms have emerged as a promising approach to OOD detection, we observe that there is a disparity between the distance measures used for OOD detection and model training. Recent studies have attempted to mitigate this shortcoming by modifying the contrastive learning process, which is highly undesirable for foundation models. In this paper, we propose an Angular distance-based out-of-distribution detection method for Contrastive models (AEC), an OOD detection framework for foundational contrastive models based on an angular distance measure. The angular distance-based score is compliant with the standard training process and circumvents the need to modify the training process of the model. We also formulate a distance transformation and solve an optimization problem to determine an OOD score threshold value for in-distribution and out-of-distribution data. The experimental evaluation demonstrates that AEC outperforms state-of-the-art OOD detection models in terms of AUROC, FPR@95TPR, accuracy, and correct ID metrics. We obtained an overall AUROC and FPR@95TPR of 70.42 and 83.98 from the proposed algorithm, which is significantly better compared to the SOTA OOD detection algorithms.

The emergence of data-intensive applications has spurred the interest for in-memory computing using nanoscale crossbars. Flow-based in-memory computing is a promising approach for evaluating Boolean logic using the natural flow of electrical currents. While automated synthesis approaches have been developed for 2D crossbars, 3D crossbars have advantageous properties in terms of density, area, and performance. In this paper, we propose the first framework for performing flow-based computing using 3D crossbars. The framework, FLOW-3D, automatically synthesizes a Boolean function into a crossbar design. FLOW-3D is based on an analogy between BDDs and crossbars, resulting in the synthesis of 3D crossbar designs with minimal semiperimeter. A BDD with $n$ nodes is mapped to a 3D crossbar with (n + k) metal wires. The $k$ extra metal wires are needed to handle hardware-imposed constraints. Compared with the state-of-the-art synthesis tool for 2D crossbars, FLOW-3D improves semiperimeter, area, energy consumption, and latency up to 61%, 84%, 37%, and 41% on 15 Revlib benchmarks.

In-memory computing is a promising solution strategy for data-intensive applications to circumvent the von Neumann bottleneck. Flow-based computing is the concept of performing in-memory computing using sneak paths in nanoscale crossbar arrays. The limitation of the previous work is that the resulting crossbar representations have large size. In this article, we present a framework called COMPACT for mapping Boolean functions to crossbar representations with a minimal semiperimeter (the number of wordlines plus bitlines) and/or maximum dimension (the maximum of the wordlines or bitlines). The COMPACT framework is based on an analogy between binary decision diagrams (BDDs) and nanoscale memristor crossbar arrays. More specifically, nodes and edges in a BDD correspond to wordlines/bitlines and memristors in a crossbar array, respectively. The relation enables a Boolean function represented by a BDD with $n$ nodes and an odd cycle transversal of size $k$ to be mapped to a crossbar with a semiperimeter of $n+k$ . The $k$ extra wordlines/bitlines are introduced due to crossbar connection constraints, i.e., wordlines (bitlines) cannot directly be connected to wordlines (bitlines). Moreover, there exists a tradeoff between the semiperimeter and maximum dimension. Consequently, COMPACT can sometimes reduce the maximum dimension by slightly increasing the length of the semiperimeter. We also extend COMPACT to handle multioutput functions using shared BDD (SBDDs) and alignment constraints on the inputs and outputs. Compared with the state-of-the-art mapping technique, the semiperimeter and maximum dimension are reduced by 55% and 85%, respectively. The area, power consumption, and computation delay are reduced by 89%, 19%, 56%, respectively.

Processing In Memory (PIM) accelerators are promising architecture that can provide massive parallelization and high efficiency in various applications. Such architectures can instantaneously provide ultra-fast operation over extensive data, allowing real-time performance in data-intensive workloads. For instance, Resistive Memory (ReRAM) based PIM architectures are widely known for their inherent dot-product computation capability. While the performance of such architecture is essential, reliability and accuracy are also important, especially in mission-critical real-time systems. Unfortunately, the PIM architectures have a fundamental limitation in guaranteeing error-free operation. As a result, current methods must pay high implementation costs or performance penalties to achieve reliable execution in the PIM accelerator. In this paper, we make a fundamental observation of this reliability limitation of ReRAM based PIM architecture. Accordingly, we propose a novel solution--Falut Tolerant PIM or FAT-PIM, that can improve reliability for such systems significantly at a low cost. Our evaluation shows that we can improve the error tolerance significantly with only 4.9% performance cost and 3.9% storage overhead.

This paper formally models the strategic repeated interactions between a system, comprising of a machine learning (ML) model and associated explanation method, and an end-user who is seeking a prediction/label and its explanation for a query/input, by means of game theory. In this game, a malicious end-user must strategically decide when to stop querying and attempt to compromise the system, while the system must strategically decide how much information (in the form of noisy explanations) it should share with the end-user and when to stop sharing, all without knowing the type (honest/malicious) of the end-user. This paper formally models this trade-off using a continuous-time stochastic Signaling game framework and characterizes the Markov perfect equilibrium state within such a framework.

Processing in-memory paves the way for neural network inference engines. An arising challenge is to develop the software/hardware interface to automatically compile deep learning models onto in-memory computing platforms. In this paper, we observe that the data layout organization of a deep neural network (DNN) model directly impacts the model's classification accuracy. This stems from that the resistive parasitics within a crossbar introduces a dependency between the matrix data and the precision of the analog computation. To minimize the impact of the parasitics, we first perform a case study to understand the underlying matrix properties that result in computation with low and high precision, respectively. Next, we propose the XORG framework that performs data layout organization for DNNs deployed on in-memory computing platforms. The data layout organization improves precision by optimizing the weight matrix to crossbar assignments at compile time. The experimental results show that the XORG framework improves precision with up to 3.2X and 31% on the average. When accelerating DNNs using XORG, the write bit-accuracy requirements are relaxed with 1-bit and the robustness to random telegraph noise (RTN) is improved.

Resistive random-access memory (ReRAM) memristors are promising candidates for various compute in memory and flow-based computing approaches. As an alternative to traditional von Neumann computation, flow-based computing avoids serial movement of data between memory and processor. In this paper, we demonstrate arrays of 1 transistor 1 ReRAM (1T1R) to detect edges between 8 bit pixels using flow-based computing, and the effects of stochastic variation of ReRAM on edge detection outputs. Three different R-off/R-on resistance ratios (1.5:1, 2.5:1 or 28.6:1) were utilized to implement multiple flow-based edge detection computation matrices for 8 bit pixels. Edge detection was distinguishable for all R-off/R-on ratios used, for all flow-based computing matrices. However, the binary output resistance ratio of the matrices improved 3-fold when the patterned R-off/R-on ratio was increased to 28.6:1. A Gaussian simulation of ReRAM resistance variability validates the experimental data, with a correlation coefficient (r) of 0.9547. These results suggest a trade-off between the flow-based edge detection output ratio and the variability of the ReRAM resistance in R-off/R-on resistance ratio.

The training of neural networks using different deep learning frameworks may lead to drastically differing accuracy levels despite the use of the same neural network architecture and identical training hyperparameters such as learning rate and choice of optimization algorithms. Currently, our ability to build standardized deep learning models is limited by the availability of a suite of neural network and corresponding training hyperparameter benchmarks that expose differences between existing deep learning frameworks. In this paper, we present a living dataset of models and hyperparameters, called CrossedWires, that exposes semantic differences between two popular deep learning frameworks: PyTorch and Tensorflow. The CrossedWires dataset currently consists of models trained on CIFAR10 images using three different computer vision architectures: VGG16, ResNet50 and DenseNet121 across a large hyperparameter space. Using hyperparameter optimization, each of the three models was trained on 400 sets of hyperparameters suggested by the HyperSpace search algorithm. The CrossedWires dataset includes PyTorch and Tensforflow models with test accuracies as different as 0.681 on syntactically equivalent models and identical hyperparameter choices. The 340 GB dataset and benchmarks presented here include the performance statistics, training curves, and model weights for all 1200 hyperparameter choices, resulting in 2400 total models. The CrossedWires dataset provides an opportunity to study semantic differences between syntactically equivalent models across popular deep learning frameworks. Further, the insights obtained from this study can enable the development of algorithms and tools that improve reliability and reproducibility of deep learning frameworks. The dataset is freely available at https://github.com/maxzvyagin/crossedwires through a Python API and direct download link.

Artificial intelligence and machine learning (AI/ML) techniques are fueling a revolution in how scientific experiments are designed, implemented and automated. Specifically, increasing high-bandwidth instruments coupled to new hardware and software systems can significantly improve the throughput of experimental results, while AI/ML techniques can provide insights into novel science and theories that were hitherto inaccessible. Despite recent progress in such “self-driving labs”, these automated platforms are susceptible to adversarial attacks as well as more traditional cybersecurity attacks. Using a motivating example of an automated approach to design anti-microbial peptides (AMP), our position paper seeks to demonstrate how a lack of adversarial robustness of AI systems such as protein folding networks may affect the execution of such experimental workflows. We highlight important problems in adversarial robustness that may need to be resolved in order to establish a trustworthy and safe AI -driven AMP synthesis system.

We design and fabricate a flow-based circuit for edge detection in images that exploits device-level parallelism in nanoscale memristor crossbars. In our approach, a corpus of human-labeled edges in BSDS500 images is used to learn an edge detection function with ternary values: true, false, and don’t-care. A Boolean crossbar design implementing an approximation of this ternary function using in-memory flow-based computing is then obtained using a massively parallel simulated annealing search executed on GPUs. We demonstrate the success of our approach by fabricating the memristor circuit on a 300nm wafer platform using a custom 65nm CMOS/ReRAM process technology. We demonstrate that our flow-based computing approach is either faster, more energy-efficient or produces fewer incorrect edges than other competing approaches. We show that our design has power and area requirements that are 3.3x and 2.5x lower, respectively, than the previous state-of-the-art.

We present a machine learning based approach for automatically synthesizing a memristor crossbar design from the specification of a Boolean formula. In particular, our approach employs deep neural networks to explore the design space of crossbar circuits and conjecture the design of an approximately correct crossbar. Then, we employ simulated annealing to obtain the correct crossbar design from the approximately correct design. Our experimental investigations show that the deep learning system is able to prune the search space to less than \\(0.0000011\\%\\) of the original search space with high probability; thereby, making it easier for the simulated annealing algorithm to identify a correct crossbar design. We automatically design an adder, subtractor, comparator, and parity circuit using this combination of deep learning and simulated annealing, and demonstrate their correctness using circuit simulations. We also compare our approach to vanilla simulated annealing without the deep learning component, and show that our approach needs only 6.08% to 69.22% of the number of circuit simulation queries required by simulated annealing alone.

We demonstrate how a target model's generalization gap leads directly to an effective deterministic black box membership inference attack (MIA). This provides an upper bound on how secure a model can be to MIA based on a simple metric. Moreover, this attack is shown to be optimal in the expected sense given access to only certain likely obtainable metrics regarding the network's training and performance. Experimentally, this attack is shown to be comparable in accuracy to state-of-art MIAs in many cases.

Sneak paths in nanoscale memristor crossbars have traditionally been viewed as a problem in the use of memristor crossbars as non-volatile replacements of traditional volatile RAM memories. We show that the sneak paths in a memristor crossbar can be employed to perform computation that exploits device-level parallelism. Our computation can be performed in the memory and does not require data to be moved between a processor and a memory unit – thereby, avoiding the von Neumann bottleneck. We demonstrate the potential of our approach by applying it to a basic problem in computer vision: edge detection in an image. Our results show that the flow-based computing approach on nanoscale memristor crossbars can be used to obtain high-quality approximations of edge detection. We have synthesized multiple $8\times 8$ crossbar circuits for this purpose – a single crossbar circuit for detecting edges between all possible pixel pairs with ~85% accuracy, and another family of input-aware crossbars with higher performance over real-world images. The family of input-aware crossbars together performs approximate edge detection for a subset of pixel pairs obtained from analyzing the BSD500 database, and the resultant images are of a quality comparable to exact edge detection.