The development of Convolutional Neural Network (CNN) architectures has traditionally relied on ad hoc and brute force approaches, often with limited justification for design choices. This lack of theoretical grounding has been detrimental to their broader adoption in critical domains such as healthcare, finance, and energy systems. To this end, we revisit the operation of CNNs through the lens of matched filtering, a classical signal detection technique rooted in systems science. Such conceptual insight provides a first-principles framework for understanding CNN functionality, enabling the interpretability of the canonical convolution-activation-pooling pipeline, and facilitating their informed design. Moreover, this perspective supports domain-aware initialisation, suggesting ways to improve architectural efficiency and accelerate convergence. It is our hope that this Lecture Note will help establish a seamless bridge between well-established signal processing principles and the often opaque methodologies of deep learning, thereby demystifying CNNs for data analytics practitioners and for educational purposes.
Backpropagation (BP) of errors is currently the backbone training algorithm for artificial neural networks (ANNs). It works by updating the network weights through gradient descent to minimize the value of a loss function, which represents the mismatch between the network’s prediction and the desired output. BP relies on the chain rule from calculus to propagate the loss gradient backward through the network’s hierarchy, allowing each weight to be efficiently and precisely updated based on its contribution to the output error. However, this process constrains the weight updates at every layer to rely on a global error signal generated at the extreme of the hierarchy. By contrast, the Hebbian model of synaptic plasticity in the brain states that weight updates should be local, determined only by the activity of presynaptic and postsynaptic neurons. According to Hebb’s model, it is therefore unlikely that biological brains directly implement BP. Recently, an alternative algorithm for training ANNs called predictive coding (PC) is gaining interest, appearing as a more biologically plausible alternative that updates the network weights using only local information. Originating from Elias’s 1950s work on signal compression [1], PC was later proposed in neuroscience as a model of the visual cortex by Rao and Ballard [2]. Successively, Friston formalized it under the free energy principle (FEP) [3], [4], grounding PC within the frameworks of Bayesian inference and dynamical systems. PC weight updates rely only on presynaptic and postsynaptic information, eliminating BP’s dependence on a global error signal. Moreover, it theoretically provides features beyond those of standard BP, such as the ability to automatically scale each gradient based on the associated uncertainty. Despite these advantages, PC still faces several open challenges: iterative error minimization can slow training compared to BP, and scaling to very deep architectures remains difficult [5]. At the same time, its local learning rules and modular structure offer promising opportunities, including highly parallelizable hardware implementations, biologically plausible learning in ANNs, and new connections between artificial intelligence (AI) and neuroscience. These aspects make PC an active area of research, with both theoretical and practical questions yet to be explored. This “Lecture Notes” column offers a novel, tutorial-style introduction to PC, focusing on its formulation, derivation, and connections to well-established optimization and signal processing algorithms, such as BP and the Kalman filter (KF). It aims to provide accessible support to the existing literature, guiding readers from the mathematical foundations underlying PC to its practical implementation, including computational examples in Python using the PyTorch framework.
Provides society information that may include news, reviews or technical notes that should be of interest to practitioners and researchers.
Provides society information that may include news, reviews or technical notes that should be of interest to practitioners and researchers.
The rapid rise of LLMs over the last few years has promoted growing experimentation with LLM-driven AI tutors. However the details of implementation, as well as the benefit in a teaching environment, are still in the early days of exploration. This article addresses these issues in the context of implementation of an AI Teaching Assistant (AI-TA) using Retrieval Augmented Generation (RAG) for Trinity College Dublin?s Master?s Motion Picture Engineering (MPE) course. We provide details of our implementation (including the prompt to the LLM, and code [1](#fn-0005)), and highlight how we designed and tuned our RAG pipeline to meet course needs. We describe our survey instruments and report on the impact of the AI-TA through a number of quantitative metrics. The scale of our experiment (43 students, 296 sessions, 1,889 queries over 7 weeks) was sufficient to have confidence in our findings. Unlike previous studies, we experimented with allowing the use of the AI-TA in open-book examinations. Statistical analysis across three exams showed no performance differences regardless of AI-TA access ( p>0.05), demonstrating that thoughtfully designed assessments can maintain academic validity. Student feedback revealed that the AI-TA was beneficial (mean = 4.22/5), while students had mixed feelings about preferring it over human tutoring (mean = 2.78/5).
The celebrated iterative soft thresholding algorithm (ISTA) and its accelerated variant, fast ISTA (FISTA), are classical signal processing methods used to solve the LASSO problem [1]. This problem spans various applications, including medical imaging, direction-of-arrival estimation, astronomy, and sparse coding. Within the broader trend in signal processing of transitioning from classical model-based approaches to deep learning-based methods, the Learned ISTA (LISTA) algorithm was proposed as a way to solve the LASSO problem using deep learning while preserving the original ISTA structure [2]. LISTA learns a fast approximation to the LASSO problem by casting the weight matrices of the ISTA algorithm as learnable parameters and making them unique for each iteration, a technique now known as deep unfolding [3]. LISTA achieves superior reconstruction results in fewer iterations compared to ISTA. This is due to two main factors. First, by learning its weights, LISTA addresses potential modeling mismatches, such as imperfect knowledge of noise behavior or the forward model. Second, even with an accurately known model, the sparsifying basis might be too complex to implement efficiently using classical methods. Here, we consider a geometric interpretation to gain insight into why LISTA performs well with significantly fewer iterations (or folds) compared to ISTA. To that end, we use the fact that both models are continuous piecewise linear (CPWL) functions. Our main contributions are as follows: • We demonstrate that existing bounds on the complexity of the geometry for ISTA and LISTA are insufficient to explain their differences; their geometries must be assessed experimentally. • We introduce the concepts of expected knot density and decision density as practical metrics to evaluate the geometry of these algorithms. • We establish a lower bound on the MAP optimal decision density for sparse linear inverse problems. • We demonstrate the effect of the loss function on the geometry of LISTA. Training LISTA with an L1 norm produces fewer larger regions and a lower knot density, compared to an L2 loss, which creates many small regions and a higher knot density. • We highlight that LISTA converges faster than ISTA and, when trained with L1, produces a simpler geometry with lower knot and decision densities closer to the optimal.
In this paper, we examine the windows available in the literature, that can be employed for spectral resolution, and see whether one of those windows can be modified to get enhanced features suitable for good spectral resolution. In the next section, a brief review of windows suitable for spectral resolution is given.
The goal of this paper is to provide an overview of recent methods for handling missing data in signal processing methods, from their origins to the challenges ahead. Missing data approaches are grouped by three main categories: i) missing-data imputation, ii) estimation with missing values and iii) prediction with missing values. We focus on methodological and experimental results through specific case studies on real-world applications. Promising and future research directions, including a better integration of informative missingness, are also discussed. We believe that the proposed conceptual framework and the presentation of the main problems related to missing data will encourage researchers of the signal processing community to develop original methods for handling missing values and to deal with new applications involving missing data in an adequate manner.
This paper presents a simple trick to increase aliasing rejection in comb decimation filters, based on modified symmetrical polynomials with unit coefficients, where the middle coefficients are set to zero. Two theorems prove that the zeros of the modified polynomials are all on the unit circle, for odd and even decimation factors. The modified polynomials are used to create modified combs, thus introducing various zeros either in the vicinity or inside the comb folding bands. As a result, the folding bands become wider and with an increased alias attenuation. An additional modified comb is introduced to improve alias attenuation further. The method is illustrated with two examples -one for odd and one for even decimation factors- providing a clear understanding of its application. Finally, the corresponding structure is presented, demonstrating the design's simplicity.
Deep neural networks (DNNs) have achieved groundbreaking success in various fields. Yet, the inherent nonconvexity of their training objectives and the complexity of their architectures present challenges for optimization and theoretical understanding. Our article examines the hidden convex structure within DNNs by drawing upon frameworks from sparse signal processing, specifically least absolute shrinkage and selection operator (LASSO), group LASSO, and nuclear-norm regularized models. By leveraging these established techniques from sparse linear models, we demonstrate how convex formulations can yield global optima and facilitate the interpretability of deep networks. Furthermore, we explore convolutional neural networks (CNNs), transformers, diffusion models, and generative adversarial networks from this convex regularization framework. We show how hypercomplex structures, such as geometric algebra, emerge in this analysis, offering new insights into the generalization to unseen data. We aim to engage the signal processing community in exploring these convex perspectives to better understand deep learning and its practical applications.
The advent of deep learning has immeasurably changed the ways we process data in signal processing and machine learning. However, training and deploying modern deep learning models demand substantial computational resources, raising concerns about exorbitant training costs, GPU shortages, and heightened energy consumption. Several lines of research over the last decade have explored the emergence of low-dimensional structures during the training process, where basic elements such as weight matrices and representations tend to be approximately low rank even though not explicitly trained to be. These low-dimensional structures arise in part due to the implicit bias of the methods used to train deep networks, providing the potential to partially explain why deep models need fewer samples than the number of model parameters. This implicit low dimensionality has inspired the exploration of low-rank structures in training and fine-tuning large-scale deep learning models more efficiently. In this article, we review recent exciting advances in using low-rank structure in deep learning and aim to clarify the mathematical foundations underlying their design. Specifically, we highlight key insights from a rich line of research focused on theoretically understanding and leveraging low-rank structures in deep learning both at every iteration of training as well as at the global minimum.
Deep neural networks (DNNs) trained via gradient descent (GD) with random initialization and without any regularization enjoy good generalization performance in practice despite being highly overparametrized. To theoretically understand this puzzling phenomenon, many works on convergence analysis for GD algorithms on NNs have been developed over the last half-decade. In this article, we review these research efforts and discuss how they address three specific questions related to this puzzle. The first question is why GD finds a global minimum efficiently, which the literature has addressed by studying what level of overparametrization (width, depth, etc.) and what type of initialization lead to a benign optimization landscape along the training trajectory, facilitating a linear convergence rate of GD. The next question is why the global minimum found by GD generalizes well, which has been addressed by showing that overparametrization induces an implicit simplicity bias along the GD trajectory. More recently, it has been observed that in practice, training DNNs with far larger learning rates than theoretically permissible results in faster convergence and better generalization. This leads to the third question of why faster convergence and better generalization can be achieved with a large learning rate, which has been recently addressed by identifying a self-stabilization mechanism and implicit bias toward flat minima.
This article reviews modern optimization methods for training neural networks (NNs) with an emphasis on efficiency and scale. We present state-of-the-art optimization algorithms under a unified algorithmic template that highlights the importance of adapting to the structures in the problem. We then cover how to make these algorithms agnostic to the scale of the problem. Our exposition is intended as an introduction for both practitioners and researchers who wish to be involved in these exciting new developments.
The advent of deep learning has immeasurably changed the ways we process data in signal processing and machine learning. However, training and deploying modern deep learning models demand substantial computational resources, raising concerns about exorbitant training costs, GPU shortages, and heightened energy consumption. Several lines of research over the last decade have explored the emergence of low-dimensional structures during the training process, where basic elements such as weight matrices and representations tend to be approximately low rank even though not explicitly trained to be. These low-dimensional structures arise in part due to the implicit bias of the methods used to train deep networks, providing the potential to partially explain why deep models need fewer samples than the number of model parameters. This implicit low dimensionality has inspired the exploration of low-rank structures in training and fine-tuning large-scale deep learning models more efficiently. In this article, we review recent exciting advances in using low-rank structure in deep learning and aim to clarify the mathematical foundations underlying their design. Specifically, we highlight key insights from a rich line of research focused on theoretically understanding and leveraging low-rank structures in deep learning both at every iteration of training as well as at the global minimum.
Deep neural networks (DNNs) have achieved groundbreaking success in various fields. Yet, the inherent nonconvexity of their training objectives and the complexity of their architectures present challenges for optimization and theoretical understanding. Our article examines the hidden convex structure within DNNs by drawing upon frameworks from sparse signal processing, specifically least absolute shrinkage and selection operator (LASSO), group LASSO, and nuclear-norm regularized models. By leveraging these established techniques from sparse linear models, we demonstrate how convex formulations can yield global optima and facilitate the interpretability of deep networks. Furthermore, we explore convolutional neural networks (CNNs), transformers, diffusion models, and generative adversarial networks from this convex regularization framework. We show how hypercomplex structures, such as geometric algebra, emerge in this analysis, offering new insights into the generalization to unseen data. We aim to engage the signal processing community in exploring these convex perspectives to better understand deep learning and its practical applications.
Neural networks (NNs) have transformed the field of artificial intelligence, showcasing remarkable abilities in learning complex representations, scaling to large datasets, and driving advancements across a wide range of applications. The training of these networks often involves optimizing functions that are not only nonconvex but possibly also nonsmooth, which introduces significant challenges in algorithm design and analysis. In this article, we provide an overview of recent advancements in the theoretical and computational aspects of nonsmooth optimization, thereby highlighting an area of growing importance that we refer to as “computational nonsmooth analysis.” We also demonstrate the relevance of these advancements in some representative applications. Lastly, we discuss some fundamental open questions in this area, the answers to which will impact the use and understanding of nonsmooth models in modern deep learning and signal processing.
