Lyle School of Engineering

Large mixture-of-experts (MoE) language models with 26–120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand. We present RotaryQuant, a three-axis compression system that addresses all three. Mixed-precision weight quantization assigns bit-widths by architectural role: 4-bit for dense layers, 2-bit for routed experts, and 8-bit for the shared expert whose high activation kurtosis resists aggressive compression. LRU expert offloading pages non-resident experts to disk under genuine memory pressure. The novel axis is IsoQuant, a KV cache compression method that applies a Walsh–Hadamard transform followed by block-diagonal SO(4) rotations to isotropize activation distributions before 3-bit scalar quantization, requiring O(d log d) operations and 256 stored parameters per head versus O(d^2) and 16,384 for dense rotation methods. A fused four-kernel Metal GPU pipeline performs attention directly on packed 3-bit tensors without materializing full-precision KV state—a different execution model, not just a quantization scheme. The combined system fits Gemma 4-26B-A4B and Qwen3-30B-A3B within a 16 GB budget and Nemotron-H 120B within 32 GB, running interactively at 9–19 tok/s with near-zero perplexity degradation (ΔPPL ≤ +0.0012) and 100% retrieval accuracy at 32K context.

Since its introduction, softmax attention has become the backbone of modern transformer architectures due to its expressiveness and scalability across a wide range of tasks. However, the main drawback of softmax attention is the quadratic memory requirement and computational complexity with respect to the sequence length. By replacing the softmax nonlinearity, linear attention and similar methods have been introduced to avoid the quadratic bottleneck of softmax attention. Despite these linear forms of attention being derived from the original softmax formulation, they typically lag in terms of downstream accuracy. While strong intuition of the softmax nonlinearity on the query and key inner product suggests that it has desirable properties compared to other nonlinearities, the question of why this discrepancy exists still remains unanswered. This work demonstrates that linear attention is a first-order approximation of the softmax numerator by deriving its full recurrent form. We further show empirically that the denominator's function can be effectively replaced by a simple vector norm. Using this form, each part of softmax attention can be described in the language of recurrent neural networks (RNNs). Describing softmax attention as an RNN allows for the ablation of the components of softmax attention to understand the importance of each part and how they interact. In this way, our work helps explain why softmax attention is more expressive than its counterparts. Code found at: https://github.com/gmongaras/On-the-Expressiveness-of-Softmax-Attention-A-Recurrent-Neural-Network-Perspective

With the increasing demand for sustainable and non-toxic alternatives, bio-based plasticizers derived from renewable sources are being developed as environmentally friendly replacements for conventional synthetic plasticizers such as phthalate esters, adipates, trimellitates, benzoates, sebacates, etc. This study investigated the extraction of solid plasticizers from the leaves of the abundantly available Millettia pinnata plant (MPL). It was chemically treated through processes including phytoremediation, slow pyrolysis, alkylation, and filtration to extract the plasticizers. Scanning electron microscopy revealed a porous, smooth surface, while atomic force microscopy further supported the morphological suitability of these materials for biofilm and composite preparation. Fourier transform infrared spectroscopy identified functional groups such as alcohol, amine, amide, hydrocarbon, alkene, and aromatic compounds, while UV analysis confirmed the presence of alcoholic, amino, and carboxyl constituents. The primary phytoconstituents detected in the MPL were molecularly docked to determine binding affinity. Thermal analysis demonstrated that the extracted plasticizer can withstand temperatures up to 267 °C. Furthermore, X-ray Diffraction analysis yielded a high crystallinity index (47.5

Peer assessment has become a prominent strategy in higher education for fostering active learning, collaboration, and critical thinking. This paper provides a critical literature-based analysis of peer assessment, comparing its effectiveness, validity, and reliability to traditional instructor-led evaluation. Empirical studies and meta-analyses show that when guided by structured rubrics, training, and anonymity, peer assessment achieves comparable performance outcomes and scoring accuracy to expert grading, while also enhancing student engagement and metacognitive development. Despite its benefits, peer assessment poses persistent challenges related to bias, reviewer inconsistency, and scalability particularly in large or online courses. In response, we propose a modular architecture for AI-supported peer assessment that integrates learning analytics, reviewer calibration, and natural language processing. The proposed architecture consists of two interdependent modules: a personalized learning module powered by explainable AI, which delivers adaptive content and supports individualized student progression; and an AI-supported peer assessment module that uses classification, clustering, and natural language processing to orchestrate equitable and high-quality peer feedback. This dual-layer approach ensures that students enter the peer review process with adequate preparation while promoting fairness and metacognitive engagement during evaluation. To promote responsible AI use, the framework integrates ethical safeguards that address data privacy, algorithmic bias, and transparency.

21st-century education demands agile learning environments that foster collaboration, critical thinking, and adaptability to prepare students for dynamic global challenges. Active learning pedagogies, such as Project-Based Learning (PPBL), align with these needs by engaging students in real-world, collaborative projects. However, traditional tools like Learning Management Systems often lack the flexibility to support industry-aligned workflows. This paper presents a novel methodology that leverages GitHub to enhance active learning within a PPBL framework. By integrating GitHub’s features: project boards to visualize learning progress, forks to provide unified templates, pull requests for efficient collaboration and peer review, and badges as a reward system, the methodology promotes agility, transparency, collaboration, and student participation. A literature review reveals that while GitHub is increasingly adopted in education, its implementation often lacks structured integration with active learning pedagogies. The proposed approach addresses this gap, offering a framework to enhance motivation, teamwork, and critical thinking. A case study in a software engineering course demonstrates improved engagement and learning outcomes, suggesting the methodology’s potential for broader educational applications.