The development of Convolutional Neural Network (CNN) architectures has traditionally relied on ad hoc and brute force approaches, often with limited justification for design choices. This lack of theoretical grounding has been detrimental to their broader adoption in critical domains such as healthcare, finance, and energy systems. To this end, we revisit the operation of CNNs through the lens of matched filtering, a classical signal detection technique rooted in systems science. Such conceptual insight provides a first-principles framework for understanding CNN functionality, enabling the interpretability of the canonical convolution-activation-pooling pipeline, and facilitating their informed design. Moreover, this perspective supports domain-aware initialisation, suggesting ways to improve architectural efficiency and accelerate convergence. It is our hope that this Lecture Note will help establish a seamless bridge between well-established signal processing principles and the often opaque methodologies of deep learning, thereby demystifying CNNs for data analytics practitioners and for educational purposes.
Backpropagation (BP) of errors is currently the backbone training algorithm for artificial neural networks (ANNs). It works by updating the network weights through gradient descent to minimize the value of a loss function, which represents the mismatch between the network’s prediction and the desired output. BP relies on the chain rule from calculus to propagate the loss gradient backward through the network’s hierarchy, allowing each weight to be efficiently and precisely updated based on its contribution to the output error. However, this process constrains the weight updates at every layer to rely on a global error signal generated at the extreme of the hierarchy. By contrast, the Hebbian model of synaptic plasticity in the brain states that weight updates should be local, determined only by the activity of presynaptic and postsynaptic neurons. According to Hebb’s model, it is therefore unlikely that biological brains directly implement BP. Recently, an alternative algorithm for training ANNs called predictive coding (PC) is gaining interest, appearing as a more biologically plausible alternative that updates the network weights using only local information. Originating from Elias’s 1950s work on signal compression [1], PC was later proposed in neuroscience as a model of the visual cortex by Rao and Ballard [2]. Successively, Friston formalized it under the free energy principle (FEP) [3], [4], grounding PC within the frameworks of Bayesian inference and dynamical systems. PC weight updates rely only on presynaptic and postsynaptic information, eliminating BP’s dependence on a global error signal. Moreover, it theoretically provides features beyond those of standard BP, such as the ability to automatically scale each gradient based on the associated uncertainty. Despite these advantages, PC still faces several open challenges: iterative error minimization can slow training compared to BP, and scaling to very deep architectures remains difficult [5]. At the same time, its local learning rules and modular structure offer promising opportunities, including highly parallelizable hardware implementations, biologically plausible learning in ANNs, and new connections between artificial intelligence (AI) and neuroscience. These aspects make PC an active area of research, with both theoretical and practical questions yet to be explored. This “Lecture Notes” column offers a novel, tutorial-style introduction to PC, focusing on its formulation, derivation, and connections to well-established optimization and signal processing algorithms, such as BP and the Kalman filter (KF). It aims to provide accessible support to the existing literature, guiding readers from the mathematical foundations underlying PC to its practical implementation, including computational examples in Python using the PyTorch framework.
Provides society information that may include news, reviews or technical notes that should be of interest to practitioners and researchers.
Provides society information that may include news, reviews or technical notes that should be of interest to practitioners and researchers.
