We propose a privacy-aware RISC-V SoC for edge inference, integrating a pipelined RV32IMC CPU, a quantized FPGA-based CNN accelerator, and a lightweight hardware security module. The accelerator employs parallel 8-bit MAC and max-pooling units. The design also features inter-layer pipelining in addition to intra-layer parallelism. This allows partial feature maps to be sent to the next layers without waiting for full completion. This approach significantly cuts down on inference latency. Moreover, we enforce security through custom CNN instructions embedding challenge-response authentication, while a noise injection unit ensures output privacy. Implemented on an Xilinx Zynq-7000 FPGA, the accelerator achieves inference latencies of 0.25 ms (MNIST) and 15.93 ms (CIFAR-10), delivering more than $10 \times$ speedup over the sequential baselines. The system delivers peak throughput of 10.56 GOPS and 2.48 GOPS, with energy efficiency reaching $60.3 \text{TOPS} / \mathrm{W}$ and $10.3 \text{TOPS} / \mathrm{W}$ for MNIST and CIFAR-10, respectively. The results demonstrate the viability of our work for secure edge AI workloads. The code is available at https://shorturl.at/hb5NU.

The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Vector-Quantization Variational Autoencoder (3D-MBQ-VAE), which combines Variational Autoencoders (VAEs) with masked token modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing a novel training strategy with full frame masking. Second, we present MotionAura, a text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture complex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, we propose a spectral transformer-based denoising network that processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-quality video generation and denoising. Lastly, we introduce a downstream task of Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our models achieve SOTA performance on a range of benchmarks. Our work offers robust frameworks for spatiotemporal modeling and user-driven video content manipulation. We will release the code, datasets, and models in open-source.

A key application of Large Language Models (LLMs) is chatbot systems, where customer queries may span heterogeneous data (structured databases, unstructured data, and web resources) and require efficient reasoning and accurate information retrieval. Often these systems struggle to deliver real-time, accurate responses for multi-source queries. This paper introduces COQO (Cost Optimal, Query Orchestration), a GUI-based tool employing a two-pipeline approach: the context-latent pipeline captures query structure, and the contentacute pipeline refines responses using specific data sources. Leveraging graph-based retrieval-augmented generation (RAG) for vector search, COQO minimizes reliance on LLMs for reasoning, enhancing cost-efficiency. We demonstrate the efficacy of the tool in terms of its cost-effectiveness handling intricate queries of a retail banking chatbot application.

The rapid growth of edge devices has revolutionized industrial AI applications, including robotics, autonomous systems, and IoT, where real-time processing is essential. These systems face the challenge of managing concurrent, high-volume workloads across resource-constrained edge devices and cloud infrastructure. A major hurdle is optimizing deep learning model deployment across edge-cloud environments in dynamic conditions, particularly where input quality and noise fluctuate under concurrent demands. This paper introduces a novel optimization framework, that addressed these challenges and dynamically selects the most suitable models from a diverse model zoo and determines optimal deployment locations (edge or cloud). The proposed framework leverages content-aware approach to minimize both communication and computation latency while considering hardware limitations and environmental factors. Using a binary linear programming (BILP) approach, our method efficiently balances model distribution of an AI pipeline, maximizing end-to-end performance. We validate this framework on a robotic AI pipeline in real-world, noise-variant environments, comparing content-aware and content-agnostic deployment strategies. Our results demonstrate significant optimization in deployment latency and system performance under high-concurrency conditions, using both content-agnostic and content-aware approaches, highlighting the framework's robustness and scalability. Additionally, we showed the effectiveness of the content-aware approach over the content-agnostic method in optimizing deployment choices and reducing latency, while maintaining the desired qualitative outcomes of the AI pipeline with different communication set up. This makes the content-aware strategy more suitable for complex, real-world environments where input quality and noise vary significantly. Overall, The proposed method presents a compelling solution for optimizing AI pipelines in edge-cloud ecosystems, offering potential for broader applications domains.

The rapid adoption of cloud computing, accelerated by the global pandemic, has increased the need for efficient cloud architecture that balances cost and performance. As organizations migrate applications to the cloud, cloud architects face challenges in managing an overwhelming number of services-often exceeding a thousand. This paper presents a novel tool designed for editable cloud architecture management that automates the optimization process. Our solution enables cloud architects to visually design and edit cloud architectures while utilizing a backend represented as a directed acyclic graph in an adjacency matrix. This structure allows for dynamic adjustments based on real-time workload predictions, moving from reactive to proactive resource management. Leveraging advanced Generative AI models, specifically Azure's GPT-4o [11], our tool identifies alternative services that can effectively replace or supplement existing ones based on functionality. By extracting relevant data from AWS documentation, we provide actionable insights on service performance and cost. We validate our approach through use cases, demonstrating the tool's effectiveness in detecting potential bottlenecks and recommending service adjustments to eliminate Service Level Agreement (SLA) violations. Our findings indicate that the tool enhances performance and reduces operational costs, empowering cloud architects to make informed, data-driven decisions. This innovative approach significantly streamlines cloud resource management, ensuring organizations can effectively navigate the complexities of their cloud environments and achieve sustained operational excellence.

Retrieval Augmented Generation (RAG) architectures have emerged as a powerful solution to enhance the accuracy and relevance of large language models (LLMs) by integrating retrieval mechanisms with generative capabilities. However, the design of an effective RAG pipeline is inherently complex, involving multiple components such as chunking strategies, embedding models, retrieval systems, and choice of LLMs. Each of these components offers numerous configuration options and the selection of the optimal combination is often a daunting task. The challenge is compounded by the need to consider trade-offs between performance, accuracy, and cost, which are not always straightforward and can vary significantly depending on the workload. In this context, we present RAGuru, an innovative tool designed to automate the design and creation of cost and latency-optimized RAG architectures. RAGuru addresses the complexities of RAG design by intelligently selecting and configuring the optimal components based on the user's specific workload requirements and also ensuring higher quality responses from RAG. By using an in-house dataset of cost and performance metrics, RAGuru ensures that the resulting architecture is cost and latency wise optimal for an use-case, while achieving high accuracy. The architecture design space choices can be fed to terraform[2] as a configuration file for automatic deployment of the cost-performance optimal RAG. We have tested RAGuru in real-world scenarios. In a particular case, the RAG generated by RAGuru, demonstrated comparable performance at approximately half the cost of a conventional RAG system, with only minimal accuracy(1) loss.

As the complexity of large language models (LLMs) increases, so does their parameter count and size. While LLMs with a substantial number of parameters yield highly accurate results, their deployment presents significant challenges even for enterprises. Existing methods for distributing transformer blocks across multiple nodes for inference are well-known; however, the responsibility for distribution typically rests with the user, often resulting in sub-optimal resource utilization. In this effort, we introduce a novel framework called ConsciousLLM, designed to self-consciously re-deploy LLMs across multiple enterprise-wide machines, by leveraging residual resources on the machines. The framework incorporates a "Self-Awareness Agent" that continuously monitors resource utilization and recalculates the optimal placement of the LLM blocks over time, thus ensuring efficient utilization of memory and compute. By dynamically redistributing transformer blocks based on real-time resource availability, the framework lowers operational costs, and improves overall system performance. We validate the efficacy of ConsciousLLM by conducting experiments with well-known open source models such as Mixtral 8x7B and LLaMA-3 (70B). Our results illustrate that capability of these models to autonomously enhance their deployment strategies, leading to optimized performance on inference tasks.

Retrieval-Augmented Generation (RAG) systems offer flexibility through modular components- chunking, embedding, retrieval, and answer generation- each with many choices, huge configuration space, and tunable hyperparameters. The choices of methods and hyperparameters affect system performance greatly, resulting in the need of optimal selection. However, selecting optimal configurations remains a challenge, especially in the case of dynamic workloads, where configurations must adapt to meet per-user SLAs without compromising costefficiency or scalability. While most research on RAG optimizations emphasizes answer quality and automatic creation of RAG systems, real-world deployments demand attention to systemlevel metrics. We propose OptiRAG, a principled and systematic framework for performance-driven RAG system design. OptiRAG combines regression-based performance modeling with constraintaware optimization to automatically select the most effective configuration across all RAG components based on workload characteristics and SLA requirements. OptiRAG adapts to changing user loads and performance constraints unlike heuristic or static approaches, ensuring robust operation across diverse deployment scenarios. We validate OptiRAG through extensive testing on multiple real-world business deployments. Across these use cases, the framework consistently identifies configurations that significantly improve system efficiency. In one high-load enterprise setting, OptiRAG achieved more than 2 X improvement in latency and approximately 150 X reduction in cost compared to a standard RAG baseline. This paper presents several case studies, demonstrating how OptiRAG enables scalable, costeffective, and SLA-compliant RAG deployments in practice.

Continuous Latent Space (CLS) and Discrete Latent Space (DLS) models, like AttnUNet and VQUNet, have excelled in medical image segmentation. In contrast, Synergistic Continuous and Discrete Latent Space (CDLS) models show promise in handling fine and coarse-grained information. However, they struggle with modeling long-range dependencies. CLS or CDLS-based models, such as TransUNet or SynergyNet are adept at capturing long-range dependencies. Since they rely heavily on feature pooling or aggregation using self-attention, they may capture dependencies among redundant regions. This hinders comprehension of anatomical structure content, poses challenges in modeling intra-class and inter-class dependencies, increases false negatives and compromises generalization. Addressing these issues, we propose L2GNet, which learns global dependencies by relating discrete codes obtained from DLS using optimal transport and aligning codes on a trainable reference. L2GNet achieves discriminative on-the-fly representation learning without an additional weight matrix in self-attention models, making it computationally efficient for medical applications. Extensive experiments on multi-organ segmentation and cardiac datasets demonstrate L2GNet's superiority over state-of-the-art methods, including the CDLS method SynergyNet, offering an novel approach to enhance deep learning models' performance in medical image analysis.

Transformer-based Large Language Models (LLMs) power applications from virtual assistants and code generation to scientific discovery. As their capabilities grow, they are used for emerging use cases such as offline AI copilots, on-device personalization, and edge inference. This demands efficient deployment of fine-tuned LLMs or compact Small Language Models (SLMs) on resource-constrained edge devices. However, deploying LLMs on the edge presents a significant hardware design challenge. The diversity in model architectures, input/output token lengths, and batch sizes leads to widely varying compute and memory demands. Moreover, design parameters like systolic array sizes, vector lengths, data widths, and operating frequencies drastically affect energy consumption and latency. While hardware-aware quantization is often adopted for power-performance gains, determining the optimal hardware configuration that meets tight power and performance budgets remains a non-trivial task, especially when the design space spans millions of possible configurations. Exhaustive exploration is computationally prohibitive and delays time to market. We introduce Architecture-Tuner (ArchTune), a lightweight analytical framework that predicts power, latency and energy consumption for RISC-V-based accelerators featuring systolic arrays and vector processing units (VPUs). Given an LLM workload, ArchTune rapidly estimates energy across millions of configurations using calibrated analytical models, eliminating the need for exhaustive simulations. ArchTune achieves $R^{2}=99.42 \%$ with $\mathbf{1 0. 4 1 \%}$ MAPE for systolic arrays and 8.2% MAPE for VPUs. By combining these models with systematic latency and memory analysis, ArchTune empowers early-stage design-space exploration, enabling designers to select energy-efficient hardware tailored for specific LLM workloads on edge platforms.

This paper introduces a highly scalable in-memory computing architecture for implementing (1) $M$-operand, $N$-bit Boolean functions, viz., AND/NAND/NOR/OR, (2) any arbitrary Boolean function expressed as the sum of products (e.g., F=AB'CD+ABC'D'), (3) $N$ parallel 2-bit XOR operations. Our technique performs operations using a modified 9T SRAM cell-based peripheral circuitry near the SRAM array. Unlike conventional 9T SRAM designs that use the same bitlines for both read and write operations, our design introduces dedicated read lines to separate the read and write paths. This separation improves efficiency and allows in-memory implementation of complex Boolean operations. To implement $M$ -operand, $N$ -bit AND/OR operations, the periphery circuit has $N$ compute blocks, each with $M$ columns. The compute block stores the input operands along with configuration bits to choose the correct operation. An equality checker circuit compares the input operands with configuration bits to obtain original or complemented input. The outputs of equality checkers of different columns are connected to a product line, and by DeMorgan's theorem, either AND/NAND or NOR/OR can be realized. Using TSMC 65nm PDK in Cadence Virtuoso, we have demonstrated 64-bit AND/NAND/OR/NOR operations and 64 parallel 2-bit XOR operations. At 0.7V VDD, our design consumes 24.5fJ/bit and operates at 660MHz for XOR operations and 384MHz for other operations.

The rapid adoption of deep learning in real-time industrial applications necessitates efficient deployment across edge-cloud infrastructures. However, deploying large deep neural networks (DNNs) entirely on edge devices is often impractical due to strict latency, energy, and memory constraints. To address this challenge, we propose ELM-Split, a network-aware edge-cloud partitioning mechanism that determines the optimal split point in a DNN by formulating a constrained optimization problem. This formulation considers per-layer latency, energy consumption, memory usage, and activation footprint. ELM-Split jointly maximizes the number of layers that can be executed on the edge and rest of the layers on cloud while maintaining edgeside energy consumption within allowable limits and memory budgets and satisfying end-to-end latency budget. It dynamically adapts to split the model varying network conditions to make intelligent partitioning decisions with in the constraints. In extensive evaluations on an XR-based EgoMono4D use case, ELM-Split achieves up to $1.57 \times$ lower end-to-end latency and $7.12 \times$ energy reduction compared to edge-only deployment under realistic 5 G network conditions. This demonstrates that ELM-Split provides a scalable and constraint-aware solution for deploying deep models in real-world edge-cloud environments.

Large Language Models (LLMs) have become integral to modern business operations, especially for tasks involving reasoning over large datasets. One prominent application of LLMs is in chatbot systems, where customers provide natural language queries, often complex in nature, requiring decomposition to retrieve relevant information from various data sources. These queries may span structured databases, unstructured data, or public information from the internet, making efficient data retrieval and reasoning vital for real-time, accurate responses. In this paper, we propose two cost-efficient "Query Orchestration" approaches (Context Latent and Context Acute) to address these challenges. By leveraging graph-based retrieval-augmented generation (RAG) techniques for vector search, we optimize data retrieval while minimizing reliance on LLMs for reasoning to reduce costs. Our approach is validated through experiments on a banking use case, where we demonstrate its effectiveness in providing high-quality responses to complex queries while ensuring cost-effectiveness.

Enterprise search systems increasingly rely on natural language interfaces-such as chatbots-to support complex, information-rich user queries. These queries often require orchestrating retrieval from diverse and distributed data sources, including structured databases, unstructured documents, and external APIs. The challenges are compounded by high query volumes, data heterogeneity, and the conversational nature of input, all of which contribute to increased latency and operational costs. To address these issues, we propose a query orchestration framework that intelligently plans and executes complex queries across concurrent sessions. Our system generates optimized query execution plans, incorporates adaptive caching strategies, and schedules sub-queries based on current workload dynamics and cache state. We evaluate our framework using real-world workloads from an open-source banking chatbot and internal enterprise data. Results show that QuOrA reduces end-to-end latency by approximately $\mathbf{2 8} \%$ and boosts throughput by 50 % compared to the baseline. Query-overlap detection achieves 57 - $59 \times$ speedup depending on the optimal execution plan. Our adaptive caching strategy achieves substantial latency and cost savings by collapsing $n$ similar queries into a single execution, yielding $n$-fold improvements. These findings highlight the importance of execution-aware query orchestration in building scalable, responsive, and cost-efficient enterprise search systems for natural language interactions.

Modern deep learning (DL) applications can be decomposed into multiple components, each with distinct compute and memory requirements. These components can be optimally mapped to different hardware accelerators to maximize efficiency. The emergence of diverse accelerators such as GPUs, Tensor Processing Units (TPUs), Gaudi, and Inferentia, along with a wide range of instance configurations offered by cloud providers, has significantly expanded the design space for heterogeneous deployments. This paper investigates the potential of such heterogeneous deployment strategies for enhancing performance and optimizing cost in real-world scenarios. We present three industrial case studies: (i) the deployment of RetrievalAugmented Generation (RAG) pipelines across heterogeneous hardware; (ii) optimization of Large Language Model (LLM) workloads, including summarization and translation, based on workload-specific characteristics; and (iii) dynamic deployment of Monte Carlo simulations in financial applications under varying user demands. These case studies illustrate how workloadaware heterogeneous deployments can deliver cost-effective, scalable, and performance-optimized solutions.

Retrieval-Augmented Generation (RAG) systems integrate large language models with information retrieval components. These systems mark a shift from structured user interfaces to natural language interfaces (NLIs). This transition eliminates the predictable interaction patterns that traditional performance testing tools rely on to control content access distributions. In RAG systems, the unstructured nature of user queries hampers the ability to simulate required content access distributions; nevertheless, content access distribution remains a critical determinant of performance due to caching effects and the uneven popularity of content. To address this challenge, we make two primary contributions: (1) an optimization-driven query issuance mechanism that enables the simulation of required content access distributions during performance testing, and (2) query generation strategies that provide broad retrieval coverage of the knowledge base while reducing query generation costs by at least $\mathbf{2 0. 9 \%}$ compared to straightforward approaches. Constructing a synthetic test query set with wide coverage is essential for shaping the desired access distribution and for identifying performance bottlenecks across both frequently and infrequently retrieved content. Empirical evaluations also demonstrate that content access distributions, particularly in the presence of caching, have a substantial impact on system's performance metrics. By enabling testing across diverse operational scenarios, our framework supports robust, access distribution-aware performance evaluation. This capability is essential for performance testing of enterprisegrade RAG applications that must meet stringent service-level agreement (SLA) requirements.

The success of application migration to cloud depends on multiple factors such as achieving expected performance, optimal cost on deployment, data security etc. The application migration process starts with the architecture design, mapping technical and business specifications to the appropriate services in cloud. However, cloud vendors offer numerous services for each service type and requirement. The onus of selecting the optimal service from the pool lies with the user. Identifying an optimal service for a specific component or application requirement is a daunting task and necessitates a deep understanding of each cloud service offered. This paper introduces SuperArch, a supervised architecture design tool designed to facilitate optimal selection and configuration of cloud services. We propose utilization of Large Language Models (LLM) for extracting information from user requirements and specifications, aiding in optimal selection of cloud services. Additionally, SuperArch maps workloads to the cloud services to generate optimal configurations of the cloud service and estimate performance and cost of the entire architecture.

In recent times, orthogonal frequency-division multiplexing (OFDM)-based radar has gained wide acceptance given its applicability in joint radar-communication systems. However, realizing such a system on hardware poses a huge area and power bottleneck given its complexity. Therefore it has become ever-important to explore low-power OFDM-based radar processors in order to realize energy-efficient joint radar-communication systems targeting edge devices. This paper aims to address the aforementioned challenges by exploiting approximations on hardware for early design space exploration (DSE) of trade-offs between accuracy, area and power. We present Ellora, a DSE framework for incorporating approximations in an OFDM radar processing pipeline. Ellora uses pairs of approximate adders and multipliers to explore design points realizing energy-efficient radar processors. Particularly, we incorporate approximations into the block involving periodogram based estimation and report area, power and accuracy levels. Experimental results show that at an average accuracy loss of 0.063% in the positive SNR region, we save 22.9% of on-chip area and 26.2% of power. Towards achieving the area and power statistics, we design a fully parallel Inverse Fast Fourier Transform (IFFT) core which acts as a part of periodogram based estimation and approximate the addition and multiplication operations in it. The aforementioned results show that Ellora can be used in an integrated way with various other optimization methods for generating low-power and energy-efficient radar processors.

The rapid expansion of Large Language Models (LLMs) presents significant challenges in efficient deployment for inference tasks, primarily due to their substantial memory and computational resource requirements. Many enterprises possess a variety of computing resources-servers, VMs, PCs, laptops-that cannot individually host a complete LLM. Collectively, however, these resources may be adequate for even the most demanding LLMs. We introduce LLaMPS, a novel tool, designed to optimally distribute blocks 1 of LLMs across available computing resources within an enterprise. LLaMPS leverages the unused capacities of these machines, allowing for the decentralized hosting of LLMs. This tool enables users to contribute their machine's resources to a shared pool, facilitating others within the network to access and utilize these resources for inference tasks. At its core, LLaMPS employs a sophisticated distributed framework to allocate transformer blocks of LLMs across various servers. In cases where a model is pre-deployed, users can directly access inference results (GUI and API). Our tool has undergone extensive testing with several open-source LLMs, including BLOOM-560m, BLOOM-3b, BLOOM-7b1, Falcon 40b, and LLaMA-70b. It is currently implemented in a real-world enterprise network setting, demonstrating its practical applicability and effectiveness.

Approximate computing offers significant gains in efficiency at the cost of minor errors. In this paper, we show that since approximate computing legitimizes controlled imprecision, this very relaxation can be exploited by an adversary to insert Trojans into approximate circuits. Since the minor errors introduced by the Trojan may be indistinguishable from those introduced by approximate computing, these Trojans can easily evade detection, yet they can severely degrade the end application’s “quality of result” (QoR). By contrast, the conventional exact computing paradigm does not tolerate errors; hence, any inserted Trojan can be easily detected. Thus, we show that approximate circuits are more vulnerable to attacks, and this may nullify their efficiency advantages. We demonstrate our ideas through the two most foundational circuits, approximate adders and multipliers. We categorize the existing approximate adders and multipliers into broad families from the perspective of Trojan insertion strategies that an adversary might employ. We present a generalized framework to identify the suitable hardware Trojan insertion and masking sites within each family of approximate adders and multipliers. We also discuss the implications of these threats for a real-life application. Our work strongly emphasizes the need for better security measures and provides insights that will guide the development of robust digital systems capable of balancing the intricacies of approximation and security.