Understanding how network connectivity shapes neural representations is central to systems neuroscience. While dimensionality reduction methods uncover low-dimensional manifold structure in population recordings, a rigorous framework connecting manifold geometry to network mechanisms and information encoding remains lacking. We develop a differential geometric approach for analyzing neural manifolds in rate-based recurrent networks receiving tuned feedforward inputs. We derive expressions for the pullback metric of neural manifolds, showing how input tuning curves together with feedforward and recurrent synaptic connectivity shape manifold geometry. Critically, we establish that the Fisher information matrix at steady states also has the structure of a pullback metric, directly linking intrinsic manifold geometry to stimulus discriminability and information encoding. For noise with slow temporal correlations propagated through the network, we show that recurrent effects on information geometry cancel: Fisher information depends only on the feedforward connectivity. Thus, feedforward connectivity critically determines representational geometry. We apply our approach to the representation of space by a module of hexagonal grid cells. We first demonstrate that the representation is approximately isometric for a random distribution of grid phases. Moreover, a linear feedforward transformation can map spatially random input tuning curves into a population of hexagonal grid cells, generating a toroidal neural manifold. Thus, feedforward connectivity alone can generate structured spatial representations without requiring carefully tuned recurrent connectivity or continuous attractor dynamics. Recurrent connectivity, however, is shown to improve stimulus encoding under fast noise, thereby implementing a selective noise reduction.

To select appropriate behaviour, individuals must rely on encoding of relevant features within their environment in the context of current and past experiences. This function has been linked to goal-associated activity patterns of hippocampal principal cells. Using single-unit recordings from optogenetically identified somatostatin-expressing interneurons (SOMIs) in the dentate gyrus of head-fixed mice trained in a spatial goal-oriented reward-learning task in virtual realities, we show that SOMI activity temporally precedes reward-locations in expert mice characterized by goal-anticipatory behaviour. Predictive goal-encoding by SOMIs is lost after translocation of learned goals to novel previously unrewarded sites leading to rapid reductions in anticipatory behaviour and fast reconfiguration of SOMI activity to times after reward onset in association with reward consumption at novel goal-sites. Chemogenetic silencing of SOMIs caused a loss of memory that trained goal-sites were no longer available. Thus, our data reveal the ability of SOMIs to flexibly encode goal-locations depending on current and past experiences to bias behavioral outcomes.

State-of-the-art neural network training methods depend on the gradient of the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful derivatives, such as binary and discrete-time spiking neural networks. To overcome this problem, the activation function's derivative is commonly substituted with a surrogate derivative, giving rise to surrogate gradient learning (SGL). This method works well in practice but lacks theoretical foundation. The neural tangent kernel (NTK) has proven successful in the analysis of gradient descent. Here, we provide a generalization of the NTK, which we call the surrogate gradient NTK, that enables the analysis of SGL. First, we study a naive extension of the NTK to activation functions with jumps, demonstrating that gradient descent for such activation functions is also ill-posed in the infinite-width limit. To address this problem, we generalize the NTK to gradient descent with surrogate derivatives, i.e., SGL. We carefully define this generalization and expand the existing key theorems on the NTK with mathematical rigor. Further, we illustrate our findings with numerical experiments. Finally, we numerically compare SGL in networks with sign activation function and finite width to kernel regression with the surrogate gradient NTK; the results confirm that the surrogate gradient NTK provides a good characterization of SGL.

Information encoding in neural circuits depends on how well time-varying stimuli are encoded by neural populations. Slow neuronal timescales, noise and network chaos can compromise reliable and rapid population response to external stimuli. A dynamic balance of externally incoming currents by strong recurrent inhibition was previously proposed as a mechanism to accurately and robustly encode a time-varying stimulus in balanced networks of binary neurons, but a theory for recurrent rate networks was missing. Here, we develop a non-stationary dynamic mean-field theory that transparently explains how a tight balance of excitatory currents by recurrent inhibition improves information encoding. We demonstrate that the mutual information rate of a time-varying input increases linearly with the tightness of balance, both in the presence of additive noise and with recurrently generated chaotic network fluctuations. We corroborated our findings in numerical experiments and demonstrated that recurrent networks with positive firing rates trained to transmit a time-varying stimulus generically use recurrent inhibition to increase the information rate. We also found that networks trained to transmit multiple independent time-varying signals spontaneously form multiple local inhibitory clusters, one for each input channel. Our findings suggest that feedforward excitatory input and local recurrent inhibition-as observed in many biological circuits-is a generic circuit motif for encoding and transmitting time-varying information in recurrent neural circuits.

Neural circuits exhibit complex activity patterns, both spontaneously and evoked by external stimuli. Information encoding and learning in neural circuits depend on how well time-varying stimuli can control spontaneous network activity. We show that in firing-rate networks in the balanced state, external control of recurrent dynamics, i.e., the suppression of internally-generated chaotic variability, strongly depends on correlations in the input. A distinctive feature of balanced networks is that, because common external input is dynamically canceled by recurrent feedback, it is far more difficult to suppress chaos with common input into each neuron than through independent input. To study this phenomenon, we develop a non-stationary dynamic mean-field theory for driven networks. The theory explains how the activity statistics and the largest Lyapunov exponent depend on the frequency and amplitude of the input, recurrent coupling strength, and network size, for both common and independent input. We further show that uncorrelated inputs facilitate learning in balanced networks.

Change is ubiquitous in living beings. In particular, the connectome and neural representations can change. Nevertheless, behaviors and memories often persist over long times. In a standard model, associative memories are represented by assemblies of strongly interconnected neurons. For faithful storage these assemblies are assumed to consist of the same neurons over time. Here we propose a contrasting memory model with complete temporal remodeling of assemblies, based on experimentally observed changes of synapses and neural representations. The assemblies drift freely as noisy autonomous network activity and spontaneous synaptic turnover induce neuron exchange. The gradual exchange allows activity-dependent and homeostatic plasticity to conserve the representational structure and keep inputs, outputs, and assemblies consistent. This leads to persistent memory. Our findings explain recent experimental results on temporal evolution of fear memory representations and suggest that memory systems need to be understood in their completeness as individual parts may constantly change.

Change is ubiquitous in living beings. In particular, the connectome and neural representations can change. Nevertheless behaviors and memories often persist over long times. In a standard model, memories are represented by assemblies of strongly interconnected neurons. For faithful storage these assemblies are assumed to consist of the same neurons over time. Here we propose a contrasting memory model with complete temporal remodeling of assemblies, based on experimentally observed changes of connections and neural representations. The assemblies drift freely as spontaneous synaptic turnover or random activity induce neuron exchange. The gradual exchange allows activity dependent and homeostatic plasticity to conserve the representational structure and keep inputs, outputs and assemblies consistent. This leads to persistent memory. Our findings explain recent experimental results on the temporal evolution of fear memory representations and suggest that memory systems need to be understood in their completeness as individual parts may constantly change.

The ability of humans and animals to quickly adapt to novel tasks is difficult to reconcile with the standard paradigm of learning by slow synaptic weight modification. Here, we show that fixed-weight neural networks can learn to generate required dynamics by imitation. After appropriate weight pretraining, the networks quickly and dynamically adapt to learn new tasks and thereafter continue to achieve them without further teacher feedback. We explain this ability and illustrate it with a variety of target dynamics, ranging from oscillatory trajectories to driven and chaotic dynamical systems.

Significance Associative memories are thought to be represented by neuronal assemblies, ensembles of nerve cells with strong synaptic interconnectivity. Experiments have, however, shown that synapses can change spontaneously. Motivated by this and by experimentally observed changes of representations, we propose that assemblies that drift freely in the brain, due to noisy network activity and spontaneous synaptic changes, are the basis of associative memory. How can memories and behaviors persist despite these changes? We find that simple, teacher-free synaptic plasticity compensates the drift and thereby solves the conundrum. When such plasticity is incorporated, networks with drifting assemblies can maintain stable memories and basic computations. The mechanisms underlying drift and compensation may apply to different kinds of drifting neural representations. Change is ubiquitous in living beings. In particular, the connectome and neural representations can change. Nevertheless, behaviors and memories often persist over long times. In a standard model, associative memories are represented by assemblies of strongly interconnected neurons. For faithful storage these assemblies are assumed to consist of the same neurons over time. Here we propose a contrasting memory model with complete temporal remodeling of assemblies, based on experimentally observed changes of synapses and neural representations. The assemblies drift freely as noisy autonomous network activity and spontaneous synaptic turnover induce neuron exchange. The gradual exchange allows activity-dependent and homeostatic plasticity to conserve the representational structure and keep inputs, outputs, and assemblies consistent. This leads to persistent memory. Our findings explain recent experimental results on temporal evolution of fear memory representations and suggest that memory systems need to be understood in their completeness as individual parts may constantly change.

Jellyfish nerve nets provide insight into the origins of nervous systems, as both their taxonomic position and their evolutionary age imply that jellyfish resemble some of the earliest neuron-bearing, actively-swimming animals. Here, we develop the first neuronal network model for the nerve nets of jellyfish. Specifically, we focus on the moon jelly Aurelia aurita and the control of its energy-efficient swimming motion. The proposed single neuron model disentangles the contributions of different currents to a spike. The network model identifies factors ensuring non-pathological activity and suggests an optimization for the transmission of signals. After modeling the jellyfish’s muscle system and its bell in a hydrodynamic environment, we explore the swimming elicited by neural activity. We find that different delays between nerve net activations lead to well-controlled, differently directed movements. Our model bridges the scales from single neurons to behavior, allowing for a comprehensive understanding of jellyfish neural control of locomotion.

Networks in the brain consist of different types of neurons. Here we investigate the influence of neuron diversity on the dynamics, phase space structure, and computational capabilities of spiking neural networks. We find that already a single neuron of a different type can qualitatively change the network dynamics and that mixed networks may combine the computational capabilities of ones with a single-neuron type. We study inhibitory networks of concave leaky (LIF) and convex "antileaky" (XIF) integrate-and-fire neurons that generalize irregularly spiking nonchaotic LIF neuron networks. Endowed with simple conductance-based synapses for XIF neurons, our networks can generate a balanced state of irregular asynchronous spiking as well. We determine the voltage probability distributions and self-consistent firing rates assuming Poisson input with finite-size spike impacts. Further, we compute the full spectrum of Lyapunov exponents (LEs) and the covariant Lyapunov vectors (CLVs) specifying the corresponding perturbation directions. We find that there is approximately one positive LE for each XIF neuron. This indicates in particular that a single XIF neuron renders the network dynamics chaotic. A simple mean-field approach, which can be justified by properties of the CLVs, explains the finding. As an application, we propose a spike-based computing scheme where our networks serve as computational reservoirs and their different stability properties yield different computational capabilities.

Networks in the brain consist of different types of neurons. We here investigate the influence of neuron diversity on the dynamical stability, phase space structure, and computational capabilities of inhibitory spiking networks using mixtures of neurons with positive and negative dissipation. We find that already a single neuron of a different type may qualitatively change the dynamics of a neural network and that mixed networks can combine the computational capabilities of ones with only a single neuron type. Our networks contain leaky integrate-and-fire neurons (positive dissipation) as well as neurons with inverted sign of the leak current (negative dissipation). We first show that such networks can exhibit a balanced state of irregular, asynchronous spiking activity, if the neurons with negative dissipation are endowed with a voltage-dependent input cutoff, which may be interpreted as a simple conductance-based implementation of their synapses. We compute the voltage probability distributions and self-consistent firing rates for both types of neurons for finite size spike impacts assuming that inputs are Poissonian. The balanced state exists in a variety of neural network models and its dynamical properties such as chaoticity vary considerably. Our models generalize networks of leaky integrate-and-fire neurons where the dynamics are non-chaotic despite their irregularity. We show that switching the sign of the dissipation of a single neuron renders the entire network dynamics unstable and chaotic as indicated by a positive largest Lyapunov exponent. To further characterize the dynamics we consider the full spectrum of Lyapunov exponents and the related covariant Lyapunov vectors (CLVs), which indicate the directions in which infinitesimal perturbations grow or shrink with a rate given by the Lyapunov exponents. We find that for each ...

Experiments in various neural systems found avalanches: bursts of activity with characteristics typical for critical dynamics. A possible explanation for their occurrence is an underlying network that self-organizes into a critical state. We propose a simple spiking model for developing neural networks, showing how these may "grow into" criticality. Avalanches generated by our model correspond to clusters of widely applied Hawkes processes. We analytically derive the cluster size and duration distributions and find that they agree with those of experimentally observed neuronal avalanches.

Networks in the brain consist of different types of neurons. Here we investigate the influence of neuron diversity on the dynamics, phase space structure and computational capabilities of spiking neural networks. We find that already a single neuron of a different type can qualitatively change the network dynamics and that mixed networks may combine the computational capabilities of ones with a single neuron type. We study inhibitory networks of concave leaky (LIF) and convex anti-leaky (XIF) integrate-and-fire neurons that generalize irregularly spiking non-chaotic LIF neuron networks. Endowed with simple conductance-based synapses for XIF neurons, our networks can generate a balanced state of irregular asynchronous spiking as well. We determine the voltage probability distributions and self-consistent firing rates assuming Poisson input with finite size spike impacts. Further, we compute the full spectrum of Lyapunov exponents (LEs) and the covariant Lyapunov vectors (CLVs) specifying the corresponding perturbation directions. We find that there is approximately one positive LE for each XIF neuron. This indicates in particular that a single XIF neuron renders the network dynamics chaotic. A simple mean-field approach, which can be justified by properties of the CLVs, explains the finding. As an application, we propose a spike-based computing scheme where our networks serve as computational reservoirs and their different stability properties yield different computational capabilities.

Autonomous, randomly coupled, neural networks display a transition to chaos at a critical coupling strength. Here, we investigate the effect of a time-varying input on the onset of chaos and the resulting consequences for information processing. Dynamic mean-field theory yields the statistics of the activity, the maximum Lyapunov exponent, and the memory capacity of the network. We find an exact condition that determines the transition from stable to chaotic dynamics and the sequential memory capacity in closed form. The input suppresses chaos by a dynamic mechanism, shifting the transition to significantly larger coupling strengths than predicted by local stability analysis. Beyond linear stability, a regime of coexistent locally expansive but nonchaotic dynamics emerges that optimizes the capacity of the network to store sequential input.

Perception seems so simple.I look out of the window to see houses, trees, people walking past, the sky above, the grass below.I hear birds in the trees, cars going past, the distant sound of an alarm.The world is full of objects that make their presence known to me through my senses -what could be more simple?Yet the efficacy of perceptual experience hides a host of questions for which we do not yet have the answers.Information reaching our senses is generally incomplete, ambiguous, distributed in space and time and not neatly sorted according to its source, so a key function of our perceptual systems is to discover the likely causes of our sensations.Perception as inference or hypothesis testing, formalised in the predictive coding theory, offers an attractive framework for exploring these issues.From this perspective, regularities or patterns provide perceptual systems with some traction, allowing the formation of expectations and a basis for decomposing the world into discrete objects.But in the dynamic world which we inhabit, object representations must be similarly dynamic, and need to form and dissolve, dominate and yield, in a way that facilitates veridical perception.In this talk I will discuss auditory scene analysis in the context of predictive coding using experimental data, exemplar models, and the phenomenon of perceptual multistability.

Noise is ubiquitous in neural systems due to intrinsic stochasticity or external drive. For deterministic dynamics, neural networks of randomly coupled units display a transition to chaos at a critical coupling strength. Here, we investigate the effect of additive white noise on the transition. We develop the dynamical mean-field theory yielding the statistics of the activity and the maximum Lyapunov exponent. A closed form expression determines the transition from the regular to the chaotic regime. Noise suppresses chaos by a dynamic mechanism, shifting the transition to significantly larger coupling strengths than predicted by local stability analysis. The decay time of the autocorrelation function does not diverge at the transition, but peaks slightly above the critical coupling strength.

Neural networks of the brain form one of the most complex systems we know. Many qualitative features of the emerging collective phenomena, such as correlated activity, stability, response to inputs, chaotic and regular behavior, can, however, be understood in simple models that are accessible to a treatment in statistical mechanics, or, more precisely, classical statistical field theory. This tutorial presents the fundamentals behind contemporary developments in the theory of neural networks of rate units that are based on methods from statistical mechanics of classical systems with a large number of interacting degrees of freedom. In particular we will focus on a relevant class of systems that have quenched (time independent) disorder. In neural networks, the main source of disorder arises from random synaptic couplings between neurons. These systems are in many respects similar to spin glasses. The tutorial therefore also explains the methods for these disordered systems as far as they are applied in neuroscience. The presentation consists of two parts. In the first part we introduce stochastic differential equations in the Martin - Siggia - Rose - De Dominicis - Janssen path integral formalism. In the second part we employ this language to derive the dynamic mean-field theory for deterministic random networks, the basis of the seminal work by Sompolinsky, Crisanti, Sommers 1988, as well as a recent extension to stochastic dynamics.