Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.

Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments

Attention mechanisms, particularly softmax attention, have been instrumental in the success of transformer-based models such as GPT. However, the quadratic memory complexity of softmax attention with respect to sequence length poses significant challenges for processing longer sequences. We introduce Cottention, a novel attention mechanism that replaces the softmax operation with cosine similarity. By leveraging the properties of cosine similarity and rearranging the attention equation, Cottention achieves native linear memory complexity with respect to sequence length, making it inherently more memory-efficient than softmax attention. We demonstrate that Cottention can be reformulated as a recurrent neural network (RNN) with a finite hidden state, allowing for constant memory usage during inference. We evaluate Cottention on both the bidirectional BERT and causal GPT tasks, demonstrating comparable performance to softmax attention while significantly reducing memory requirements. To ensure efficient computation, we develop a custom CUDA kernel for Cottention. Our results show that Cottention is a promising alternative to softmax attention, enabling the processing of longer sequences without sacrificing performance, due to its native linear memory complexity and ability to maintain a constant memory footprint during inference. We provide code at https://github.com/gmongaras/Cottention_Transformer

This paper shows how to use Large Language Models (LLMs) and DSPy prompt engineering to bridge the gap between expert knowledge and user natural language inputs. This approach improves radar system usability by leveraging the DSPy ReAct module and MIPROv2 prompt optimization to facilitate the dynamic generation of radar technical parameters.

In this work, we propose an alternative method for generating the MUSIC (Multiple Signal Classification) pseudospectra by utilizing the spectral properties of Cayley graphs. Sensor readings are represented as multi-valued functions, which are used to color the edges connecting nodes representing symmetry permutation group elements. This graph representation yields decorrelating spectral information, enabling the effective separation of the signal subspace from the noise subspace required by the MUSIC algorithm. The proposed method is applied to a direction-of-arrival estimation task, demonstrating its ability to generate MUSIC pseudospectra for signal processing applications.

In education, analyzing student oral reading is critical to reading comprehension, vocabulary development, and fluency. Automating these assessments is a crucial part of language research however, researchers often use a separate model for each assessment. Transferring knowledge between assessments can make a model more reliable and robust. In this work, we introduce a unified model comprising a contrastive self-supervised embedding model and a transfer learning suffix network. The embedding model is trained on oral reading fluency data, generating student voice embeddings. Subsequently, the transfer learning network employs these embeddings as features to estimate oral reading prosody scores. Our model demonstrates high agreement with human raters, with a reliability of 0.55 (78% accuracy) on unseen passages compared to 0.63 between humans. This generalization suggests that the model can provide robust representations across various oral reading assessment tasks.

Within the context of learning, there poses difficulty when objectively measuring human performance. In this work, we investigate the evaluation of human performance via its relation to the individual's mental capacity by classification of cognitive load within the domain of aviation. By utilizing a mixed virtual and physical flight simulation environment in conjunction with biometric sensing, we create and evaluate the predictive capabilities of a Joint-Embedding Predictive Architecture (JEPA) and compare the architecture and results to traditional methods for transfer learning and domain adaptation. We find that our JEPA inspired architecture can achieve more than 70% accuracy of cognitive workload, compared to the 63% and 56% accuracies of traditional transfer learning methods. Through this foundation, we have made advancements in multi-modal and multi-task learning to classify various features across numerous pilots, operators, and novices within aviation. Our predictive model can automate the evaluation of cognitive load, enabling creation of generalizing features even when labeled examples are scarce.

Traditional statistical optimization-based state estimation (DSSE) algorithms rely on detailed grid parameters and mathematical assumptions of all possible uncertainties. Furthermore, random data missing due to communication failures, congestion, and cyberattacks, makes these methods easily infeasible. Inspired by recent advances in digital twins (DTs), this paper proposes an interactive attention-based DSSE model for robust grid monitoring by integrating three core components: physical entities, virtual modeling, and data fusion. To enable robustness against various data missing in heterogeneous measurements, we first propose physics-informed data augmentation and transfer. Moreover, a state-of-the-art attention-based spatiotemporal feature learning is proposed, followed by a novel cross-interaction feature fusion for robust voltage estimation. A case study in a real-world unbalanced 84-bus distribution system with raw data validates the accuracy and robustness of the proposed DT model in estimating voltage states, with random locational, arbitrary ratios (up to 40% of total measurements) of data missing.

Electronic Health Record (EHR) data encompasses diverse modalities-text, images, and medical codes-that are vital for clinical decision-making. To process these complex data, multimodal AI (MAI) has emerged as a powerful approach for fusing such information. However, most existing MAI models optimize for better prediction performance, potentially reinforcing biases across patient subgroups. Although bias reduction techniques for multimodal models have been proposed, the individual strengths of each modality and their interplay in both reducing bias and optimizing performance remain underexplored. In this work, we introduce FAME (Fairness-Aware Multimodal Embeddings), a framework that explicitly weights each modality according to its fairness contribution. FAME optimizes both performance and fairness by incorporating a combined loss function. We leverage the Error Distribution Disparity Index (EDDI) to measure fairness across subgroups and propose a sign-agnostic aggregation method to balance fairness across subgroups, ensuring equitable model outcomes. We evaluate FAME with BEHRT and BioClinicalBERT, combining structured and unstructured EHR data, and demonstrate its effectiveness in performance and fairness compared to other baselines across multiple EHR prediction tasks.

This study proposes and evaluates a fully decentralized Q-Learning framework for dynamic optimization of unmanned aerial system array geometries to reduce emitter localization error. By combining decentralized multi-agent Q-learning with adjustments to the array geometry based on real-time environmental feedback, signal time-difference of arrival (TDOA), and reduction of the spherical error probable (SEP) of the emitter location, the system develops a consensus policy to optimize sensor positions and converge on the true location of a sensed emitter with minimized localization errors in complex multipath environments. The consensus policy allows scalability to larger drone network sizes, further enhancing localization accuracy without additional training. Simulation results confirm the viability of the system to optimize its geometry to improve the accuracy of localization and show the scalability with a shared policy. Our results demonstrate the opportunity provided through intelligent dynamic repositioning of the UAS array to enhance localization performance in dense urban settings, offering practical approaches for effective performance in active operations of surveillance or search and rescue missions.

This paper introduces an adversarial framework that leverages two Large Language Models (LLMs) via prompt engineering to enhance phishing detection. One LLM functions as a generator, producing sophisticated phishing emails that mimic legitimate communications, while the other serves as a discriminator, detecting and classifying these emails and providing detailed reasoning for its decisions. By dynamically refining prompts based on adversarial interactions, this framework not only improves detection accuracy but also educates users on phishing indicators-helping reduce cognitive biases. Our results demonstrate a robust, adaptive defense against increasingly complex cyber threats.

We propose an attention-based multiple instance classification model (AMIC) to conduct interpretable word-level sentiment analysis (SA) using only document sentiment labels. The word-level SA adds more interpretability compared to other models while maintaining competitive performance at the document level. Furthermore, we decompose our model into interpretable outputs that provide context weighting, indication of word neutrality, and negation. This structure provides insights on how context influences sentiment and the inner workings in the model’s decision-making process. AMIC is built on a straightforward modeling framework (i.e., multiple instance classification model) which incorporates blocks of self-attention and positional encoded self-attention to achieve competitive prediction performance. The architecture is transparent yet effective at conducting interpretable SA. Model performance is reported on two document sentiment classification datasets, with extensive analysis of model interpretation.

Spiking neural networks (SNNs) are the third generation of neural networks that are biologically inspired to process data in a fashion that emulates the exchange of signals in the brain. Within the Computer Vision community SNNs have garnered significant attention due in large part to the availability of event-based sensors that produce a spatially resolved spike train in response to changes in scene radiance. SNNs are used to process event-based data due to their neuromorphic nature. The proposed work examines the neuromorphic advantage of fusing multiple sensory inputs in classification tasks. Specifically we study the performance of a SNN in digit classification by passing in a visual modality branch (Neuromorphic-MNIST [N-MNIST]) and an auditory modality branch (Spiking Heidelberg Digits [SHD]) from datasets that were created using event-based sensors to generate a series of time-dependent events. It is observed that multi-modal SNNs outperform unimodal visual and unimodal auditory SNNs. Furthermore, it is observed that the process of sensory fusion is insensitive to the depth at which the visual and auditory branches are combined. This work achieves a 98.43 SHD dataset using a multimodal SNN that concatenates the visual and auditory branches at a late depth.

Abstract Recent geophysical studies have highlighted the potential utility of integrating both seismic and infrasound data to improve source characterization and event discrimination efforts. However, the influence of each of these data types within an integrated framework is not yet well‐understood by the geophysical community. To help elucidate the role of each data type within a merged structure, we develop a neural network which fuses seismic and infrasound array data via a gated multimodal unit for earthquake‐explosion discrimination within the Korean Peninsula. Model performance is compared before and after adding the infrasound branch. We find that the seismoacoustic model outperforms the seismic model, with the majority of the improvements stemming from the explosions class. The influence of infrasound is quantified by analyzing gated multimodal activations. Results indicate that the model relies comparatively more on the infrasound branch to correct seismic predictions.

In this study, we demonstrate the feasibility of using physical sensors in System-on-a-Chip (SoC) for real-time security monitoring by detecting and characterizing different process types, such as file I/O, CPU/ALU-intensive tasks, network I/O and virtualization. We present models that use sensor data for binary classification to determine whether a specific process category is active or inactive. Analyzing the detection results, we determine the importance of each sensor in identifying process types and providing insights into process behaviors and sensor impacts. We developed adaptive ensemble classifiers to accommodate varying load conditions, enhancing detection accuracy across diverse operational scenarios, including regular background activities that simulate real-world conditions. Our results show effective detection of file I/O and CPU/ALU-intensive processes under various loads. Virtualization processes are accurately detected under light loads but show a moderate accuracy decline under heavier conditions. Network I/O detection faces challenges due to fewer relevant sensors. Our ability to predict process categories consistently and with high performance allows us to discover behaviors of various malicious activities. This research underlines the efficacy of sensor-based analysis for reliable and adaptable real-time process monitoring, demonstrating the feasibility of using sensor data for security purposes in various environmental conditions.

The quantum computer has become contemporary reality, with the first two-qubit machine of mere decades ago transforming into cloud-accessible devices with tens, hundreds, or -- in a few cases -- even thousands of qubits. While such hardware is noisy and still relatively small, the increasing number of operable qubits raises another challenge: how to develop the now-sizeable quantum circuits executable on these machines. Preparing circuits manually for specifications of any meaningful size is at best tedious and at worst impossible, creating a need for automation. This article describes an automated quantum-software toolkit for synthesis, compilation, and optimization, which transforms classically-specified, irreversible functions into both technology-independent and technology-dependent quantum circuits. We also describe and analyze the toolkit's application to three situations -- quantum read-only memories, quantum random number generators, and quantum oracles -- and illustrate the toolkit's start-to-finish features, from the input of classical functions to the output of technology-dependent quantum circuits. Furthermore, we illustrate how the toolkit enables research beyond circuit synthesis, including comparison of synthesis and optimization methods and deeper understanding of even well-studied quantum algorithms. As quantum hardware continues to develop, such quantum circuit toolkits will play a critical role in realizing its potential.

Multimodal machine learning, in the context of deep learning, allows a neural network to process various sources of data and combine information from each data source. However, there are an exponential number of ways in which modalities can be combined for processing which can result in large architecture design searches to inform the most optimal manner of combining data streams. To mitigate this problem, we present a way to inform the creation of multimodal machine learning convolutional neural network architectures in the domain of time series datasets. Specifically, we propose the use of time series clustering as a method for informing the creation of a model's multimodal architecture. We investigate two different approaches to this method (a Euclidean-and Granger-based approach) and demonstrate effectiveness with multiple time series datasets. We find that our proposed methods can improve a model's predictive capabilities while decreasing the training time required for the model to converge. Moreover, our method eliminates the need for a costly architecture search.

As the scale of deep learning tasks continues to expand, the generation of sufficiently large datasets has become increasingly costly and time-consuming. In particular, resource demands for manual annotations for computer vision tasks such as multi-object tracking have contributed to the growing popularity of synthetic computer vision datasets created through simulation engines. Simulations facilitate the creation of automatically annotated datasets with complete control over environmental variables that are typically uncontrollable in real-world scenarios. Leveraging this control, we generate multi-object tracking datasets isolating specific environmental variables including subject scale, camera movement, and lighting changes. Our evaluation focuses on the TrackFormer architecture, an end-to-end, transformer-based solution designed for multi-object tracking. The resulting insights into how each environmental variable affects multi-object tracking performance can guide future architectural improvements. Furthermore, our data generation process can serve as a template for evaluating deep learning architectures in simulated environments.

Automatic modulation classification is an important component in many modern aeronautical communication systems to achieve efficient spectrum usage in congested wireless environments and other communications systems applications. In recent years, numerous convolutional deep learning architectures have been proposed for automatically classifying the modulation used on observed signal bursts. However, a comprehensive analysis of these differing architectures and the importance of each design element has not been carried out. Thus, it is unclear what trade-offs the differing designs of these convolutional neural networks might have. In this research, we investigate numerous architectures for automatic modulation classification and perform a comprehensive ablation study to investigate the impacts of varying hyperparameters and design elements on automatic modulation classification accuracy. We show that a new state-of-the-art accuracy can be achieved using a subset of the studied design elements, particularly as applied to modulation classification over intercepted bursts of varying time duration. In particular, we show that a combination of dilated convolutions, statistics pooling, and squeeze-and-excitation units results in the strongest performing classifier achieving 98.9% peak accuracy and 63.7% overall accuracy on the RadioML 2018.01A dataset. We further investigate this best performer according to various other criteria, including short signal bursts of varying length, common misclassifications, and performance across differing modulation categories and modes.

The use of CMOS sensors for rotational spectroscopy is a promising, but challenging avenue for low-cost gas sensing and molecular identification. A main challenge in this approach is that practical CMOS spectroscopy samples contain various different noise sources that reduce the effectiveness of matching techniques for molecular identification with rotational spectroscopy. To help solve this challenge, we develop a software application tool that can demonstrate the feasibility and reliability of detection with CMOS sensor samples. Specifically, the tool characterizes the types of noise in CMOS sample collection and synthesizes spectroscopy files based upon existing databases of rotational spectroscopy samples gathered from other sensors. We use the software to create a large database of plausible CMOS-generated sample files of gases. This dataset is used to help evaluate spectral matching algorithms used in gas sensing and molecular identification applications. We evaluate these traditional methods on the synthesized dataset and discuss how peak finding and spectral matching algorithms can be altered to accommodate the noise sources present in CMOS sample collection.