Network states with rich dynamics and highly variable firing rates of individual neurons are prominent in experimental observations and thought to benefit complex information processing and learning. Such states have been proposed to arise from properties of network coupling, like a strong connectivity or slow synaptic dynamics. Here, we identify an alternative mechanism based on weak synaptic coupling and intrinsic cellular dynamics. We show that a switch in the cellular excitability class of action-potential generation (via a switch in the underlying mathematical bifurcation), further amplified by recurrent interactions, results in super-Poissonian spiking variability in random balanced networks. Information encoding is shifted to higher frequency bands and collective chaos in the network is enhanced when intrinsic cellular dynamics follow a saddle homoclinic orbit (HOM) bifurcation. The robust effect links the biophysics of individual neurons to collective dynamics of large random networks, highlighting the relevance of single-cell dynamics for computation in physiological and artificial networks. ### Competing Interest Statement The authors have declared no competing interest.

Nerve impulses, the currency of information flow in the brain, are generated by an instability of the neuronal membrane potential dynamics. Neuronal circuits exhibit collective chaos that appears essential for learning, memory, sensory processing, and motor control. However, the factors controlling the nature and intensity of collective chaos in neuronal circuits are not well understood. Here we use computational ergodic theory to demonstrate that basic features of nerve impulse generation profoundly affect collective chaos in neuronal circuits. Numerically exact calculations of Lyapunov spectra, Kolmogorov-Sinai-entropy, and upper and lower bounds on attractor dimension show that changes in nerve impulse generation in individual neurons moderately impact information encoding rates but qualitatively transform phase space structure. Specifically, we find a drastic reduction in the number of unstable manifolds, Kolmogorov-Sinai entropy, and attractor dimension. Beyond a critical point, marked by the simultaneous breakdown of the diffusion approximation, a peak in the largest Lyapunov exponent, and a localization transition of the leading covariant Lyapunov vector, networks exhibit sparse chaos: prolonged periods of near stable dynamics interrupted by short bursts of intense chaos. Analysis of large, more realistically structured networks supports the generality of these findings. In cortical circuits, biophysical properties appear tuned to this regime of sparse chaos. Our results reveal a close link between fundamental aspects of single-neuron biophysics and the collective dynamics of cortical circuits, suggesting that nerve impulse generation mechanisms are adapted to enhance circuit controllability and information flow.

Neuronal spiking activity in cortical circuits is often temporally structured by collective rhythms. Rhythmic activity has been hypothesized to regulate temporal coding and to mediate the flexible routing of information flow across the cortex. Spiking neuronal circuits, however, are non-linear systems that, through chaotic dynamics, can amplify insignificant microscopic fluctuations into network-scale response variability. In nonlinear systems in general, rhythmic oscillatory drive can induce chaotic behavior or boost the intensity of chaos. Thus, neuronal oscillations could rather disrupt than facilitate cortical coding functions by flooding the finite population bandwidth with chaotically-boosted noise. Here we tackle a fundamental mathematical challenge to characterize the dynamics on the attractor of effectively delayed network models. We find that delays introduce a transition to collective oscillations, below which ergodic theory measures have a stereotypical dependence on the delay so far only described in scalar systems and low-dimensional maps. We demonstrate that the emergence of internally generated oscillations induces a complete dynamical reconfiguration, by increasing the dimensionality of the chaotic attractor, the speed at which nearby trajectories separate from one another, and the rate at which the network produces entropy. We find that periodic input drive leads to a dramatic increase of chaotic measures at a the resonance frequency of the recurrent network. However, transient oscillatory input only has a moderate role on the collective dynamics. Our results suggest that simple temporal dynamics of the mean activity can have a profound effect on the structure of the spiking patterns and therefore on the information processing capability of neuronal networks.

Spiking Neural Networks (SNNs) are biologically-inspired models that are capable of processing information in streams of action potentials. However, simulating and training SNNs is computationally expensive due to the need to solve large systems of coupled differential equations. In this paper, we introduce SparseProp, a novel event-based algorithm for simulating and training sparse SNNs. Our algorithm reduces the computational cost of both the forward and backward pass operations from O(N) to O(log(N)) per network spike, thereby enabling numerically exact simulations of large spiking networks and their efficient training using backpropagation through time. By leveraging the sparsity of the network, SparseProp eliminates the need to iterate through all neurons at each spike, employing efficient state updates instead. We demonstrate the efficacy of SparseProp across several classical integrate-and-fire neuron models, including a simulation of a sparse SNN with one million LIF neurons. This results in a speed-up exceeding four orders of magnitude relative to previous event-based implementations. Our work provides an efficient and exact solution for training large-scale spiking neural networks and opens up new possibilities for building more sophisticated brain-inspired models.

Recurrent networks are widely used as models of biological neural circuits and in artificial intelligence applications. Mean-field theory has been used to uncover key properties of recurrent network models such as the onset of chaos and their largest Lyapunov exponents, but quantities such as attractor dimension and KolmogorovSinai entropy have thus far remained elusive. We calculate the complete Lyapunov spectrum of recurrent neural networks and show that chaos in these networks is extensive with a size-invariant Lyapunov spectrum and attractor dimensions much smaller than the number of phase space dimensions. The attractor dimension and entropy rate increase with coupling strength near the onset of chaos but decrease far from the onset, reflecting a reduction in the number of unstable directions. We analytically approximate the full Lyapunov spectrum using random matrix theory near the onset of chaos for strong coupling and discrete-time dynamics. We show that a generalized time-reversal symmetry of the network dynamics induces a point symmetry of the Lyapunov spectrum reminiscent of the symplectic structure of chaotic Hamiltonian systems. Temporally fluctuating input can drastically reduce both the entropy rate and the attractor dimension. We lay out a comprehensive set of controls for the accuracy and convergence of Lyapunov exponents. For trained recurrent networks, we find that Lyapunov spectrum analysis quantifies error propagation and stability achieved by different learning algorithms. Our methods apply to systems of arbitrary connectivity and highlight the potential of Lyapunov spectrum analysis as a diagnostic for machine learning applications of recurrent networks.

Training recurrent neural networks (RNNs) remains a challenge due to the instability of gradients across long time horizons, which can lead to exploding and vanishing gradients. Recent research has linked these problems to the values of Lyapunov exponents for the forward-dynamics, which describe the growth or shrinkage of infinitesimal perturbations. Here, we propose gradient flossing, a novel approach to tackling gradient instability by pushing Lyapunov exponents of the forward dynamics toward zero during learning. We achieve this by regularizing Lyapunov exponents through backpropagation using differentiable linear algebra. This enables us to "floss" the gradients, stabilizing them and thus improving network training. We demonstrate that gradient flossing controls not only the gradient norm but also the condition number of the long-term Jacobian, facilitating multidimensional error feedback propagation. We find that applying gradient flossing prior to training enhances both the success rate and convergence speed for tasks involving long time horizons. For challenging tasks, we show that gradient flossing during training can further increase the time horizon that can be bridged by backpropagation through time. Moreover, we demonstrate the effectiveness of our approach on various RNN architectures and tasks of variable temporal complexity. Additionally, we provide a simple implementation of our gradient flossing algorithm that can be used in practice. Our results indicate that gradient flossing via regularizing Lyapunov exponents can significantly enhance the effectiveness of RNN training and mitigate the exploding and vanishing gradient problem.

Information encoding in neural circuits depends on how well time-varying stimuli are encoded by neural populations. Slow neuronal timescales, noise and network chaos can compromise reliable and rapid population response to external stimuli. A dynamic balance of externally incoming currents by strong recurrent inhibition was previously proposed as a mechanism to accurately and robustly encode a time-varying stimulus in balanced networks of binary neurons, but a theory for recurrent rate networks was missing. Here, we develop a non-stationary dynamic mean-field theory that transparently explains how a tight balance of excitatory currents by recurrent inhibition improves information encoding. We demonstrate that the mutual information rate of a time-varying input increases linearly with the tightness of balance, both in the presence of additive noise and with recurrently generated chaotic network fluctuations. We corroborated our findings in numerical experiments and demonstrated that recurrent networks with positive firing rates trained to transmit a time-varying stimulus generically use recurrent inhibition to increase the information rate. We also found that networks trained to transmit multiple independent time-varying signals spontaneously form multiple local inhibitory clusters, one for each input channel. Our findings suggest that feedforward excitatory input and local recurrent inhibition-as observed in many biological circuits-is a generic circuit motif for encoding and transmitting time-varying information in recurrent neural circuits.

We present a novel approach to use curricula to identify principles by which a system learns. Previous work in curriculum learning has focused on how curricula can be designed to improve learning of a model on particular tasks. We consider the inverse problem: what can a curriculum tell us about how a learning system acquired a task? Using recurrent neural networks (RNNs) and models of common experimental neuroscience tasks, we demonstrate that curricula can be used to differentiate learning principles using target-based and a representation-based loss functions as use cases. In particular, we compare the performance of RNNs using target-based learning rules versus those using representational learning rules on three different curricula in the context of two tasks. We show that the learned state-space trajectories of RNNs trained by these two learning rules under all curricula tested are indistinguishable. However, by comparing learning times during different curricula, we can disambiguate the learning rules and challenge traditional approaches of interrogating learning systems. Although all animals in neuroscience lab settings are trained by curriculum-based procedures called shaping, almost no behavioral or neural data are collected or published on the relative successes or training times under different curricula. Our results motivate the systematic collection and curation of data during shaping by demonstrating curriculum learning in RNNs as a tool to probe and differentiate learning principles used by biological systems, over conventional statistical analyses of learned state spaces.

Neural circuits exhibit complex activity patterns, both spontaneously and evoked by external stimuli. Information encoding and learning in neural circuits depend on how well time-varying stimuli can control spontaneous network activity. We show that in firing-rate networks in the balanced state, external control of recurrent dynamics, i.e., the suppression of internally-generated chaotic variability, strongly depends on correlations in the input. A distinctive feature of balanced networks is that, because common external input is dynamically canceled by recurrent feedback, it is far more difficult to suppress chaos with common input into each neuron than through independent input. To study this phenomenon, we develop a non-stationary dynamic mean-field theory for driven networks. The theory explains how the activity statistics and the largest Lyapunov exponent depend on the frequency and amplitude of the input, recurrent coupling strength, and network size, for both common and independent input. We further show that uncorrelated inputs facilitate learning in balanced networks.

Neuronal activity in the central nervous system varies strongly in time and across neuronal populations. It is a longstanding proposal that such fluctuations generically arise from chaotic network dynamics. Various theoretical studies predict that the rich dynamics of rate models operating in the chaotic regime can subserve circuit computation and learning. Neurons in the brain, however, communicate via spikes and it is a theoretical challenge to obtain similar rate fluctuations in networks of spiking neuron models. A recent study investigated spiking balanced networks of leaky integrate and fire (LIF) neurons and compared their dynamics to a matched rate network with identical topology, where single unit input-output functions were chosen from isolated LIF neurons receiving Gaussian white noise input. A mathematical analogy between the chaotic instability in networks of rate units and the spiking network dynamics was proposed. Here we revisit the behavior of the spiking LIF networks and these matched rate networks. We find expected hallmarks of a chaotic instability in the rate network: For supercritical coupling strength near the transition point, the autocorrelation time diverges. For subcritical coupling strengths, we observe critical slowing down in response to small external perturbations. In the spiking network, we found in contrast that the timescale of the autocorrelations is insensitive to the coupling strength and that rate deviations resulting from small input perturbations rapidly decay. The decay speed even accelerates for increasing coupling strength. In conclusion, our reanalysis demonstrates fundamental differences between the behavior of pulse-coupled spiking LIF networks and rate networks with matched topology and input-output function. In particular there is no indication of a corresponding chaotic instability in the spiking network.

Cortical neurons operate within recurrent neuronal circuits. Dissecting their operation is key to understanding information processing in the cortex and requires transparent and adequate dynamical models of circuit function. Convergent evidence from experimental and theoretical studies indicates that strong feedback inhibition shapes the operating regime of cortical circuits. For circuits operating in inhibition-dominated regimes, mathematical and computational studies over the past several years achieved substantial advances in understanding response modulation and heterogeneity, emergent stimulus selectivity, inter-neuron correlations, and microstate dynamics. The latter indicate a surprisingly strong dependence of the collective circuit dynamics on the features of single neuron action potential generation. New approaches are needed to definitely characterize the cortical operating regime.

The prevailing explanation for the irregularity of spike sequences in the cerebral cortex is a dynamic balance of excitatory and inhibitory synaptic inputs - the socalled balanced state [1]. Nevertheless its statistical properties are well described by a mean field theory that is independent of the single neuron dynamics, its dynamics is far from being understood. Recently it was found that the stability of the balanced state dynamics depends strongly on the detailed underlying dynamics of individual neurons. Inhibitory networks of leaky integrate-and-fire neurons show stable chaos [2,3], while a balanced network of neurons with an active spike generation mechanism exhibits deterministic extensive chaos [4]. Previous studies of the dynamics of the balanced state used random (ErdŐs-Renyi) networks. We extended this analysis to arbitrary network topologies and analyzed the entropy production in small world topologies [5], ring networks [6], clustered networks [7], multi-layered networks [8] and topologies with different frequencies of certain network motifs [9]. We derived an analytical expression for the single spike Jacobian containing elements of the coupling matrix, which enabled us to calculate the full Lyapunov spectrum for any desired topology. Using a single neuron model in which action potential onset rapidness [10] and synaptic time constant are adjustable, we simulated the dynamics in numerically exact event-based simulations and calculated Lyapunov spectra, entropy production rate and attractor dimension for a variety of connectivities. We found that the importance of the internal single neuron dynamics for the network stability persists in different topologies. While the Entropy production and Attractor Dimension in clustered [7] and ring networks was very similar to random networks, these dynamical properties were changed substantially when introducing second order network motifs or a small world topology.

Information is processed in the brain by the coordinated activity of large neural circuits. Yet, we are still only starting to understand how this high-dimensional complex system gives rise to functions such as processing sensory information, making decisions and controlling behavior. Technological advances such as optogenetics and cellular resolution imaging provide tools to measure and manipulate the activity of many neurons simultaneously. These developments open novel avenues for the interplay of theory and experiment in neuroscience and foster the development of mathematical approaches for the systematic dissection and understanding of cortical information processing. This will undoubtedly allow more systematic and comprehensive insights into the brain's structure, function, dynamics, and plasticity. But given the complexity of neural network dynamics, it is not yet clear to what extent this will also give rise to a better conceptual and quantitative understanding of principles underlying neural circuit information processing. Depending on the specific question, we might need a diversity of theoretical concepts and perspectives. Among these are both mechanistic bottom-up approaches which assemble simplified well-understood units into circuits giving rise to less-understood network dynamics and normative top-down approaches, starting for example from information theoretic, geometric or evolutionary constraints to infer how computations should be performed. How information is encoded, processed and transmitted by neural circuits is intimately related to their collective network dynamics. Therefore, it is desirable to better understand how different factors shape the patterns of activity across neural populations. Prominent factors that shape circuit dynamics include single-cell properties, synaptic features, network topology and external input statistics. In this thesis, we develop novel numerical and analytical techniques from dynamical systems, stochastic processes and information theory to characterize the evoked and spontaneous dynamics and phase space organization of large neural circuit models. Our target is to determine how biophysical properties of neurons and network parameters influence information transmission. We investigate the role and relevance of single-cell properties in the collective network dynamics and study how the statistics of external input spike trains affect the chaoticity and reliability of balanced target circuits. By varying the statistics of the streams of input spike trains and investigating the scaling of properties of the collective dynamics with different network parameters, we identify key parameters that regulate information transmission and the ability to control the activity states in a driven network.