Current neuroscience research is often limited to testing predetermined hypotheses and post hoc analysis of already collected data. Adaptive experimental designs, in which modeling drives ongoing data collection and selects experimental manipulations, offer a promising alternative. However, such adaptive paradigms require tight integration between software and hardware under real-time constraints. We introduce improv, a software platform for flexible integration of modeling, data collection, analysis pipelines, and live experimental control. We demonstrate both in silico and in vivo how improv enables efficient experimental designs for discovery and validation across various model organisms and data types. We used improv to orchestrate real-time behavioral analyses, rapid functional typing of neural responses via calcium imaging, optimal visual stimulus selection, and model-driven optogenetic photostimulation of visually responsive neurons in the zebrafish brain. Together, these results demonstrate the power of improv to integrate modeling with data collection and experimental control to achieve next-generation adaptive experiments.

Light-sheet microscopes enable rapid high-resolution imaging of biological specimens; however, biological processes span spatiotemporal scales. Moreover, long-term phenotypes are often instigated by rare or fleeting biological events that are difficult to capture with a single imaging modality. Here, to overcome this limitation, we present smartLLSM, a microscope that incorporates artificial intelligence-based instrument control to autonomously switch between epifluorescent inverted imaging and lattice light-sheet microscopy (LLSM). We apply this approach to two unique processes: cell division and immune synapse formation. In each context, smartLLSM provides population-level statistics across thousands of cells and autonomously captures multicolor three-dimensional datasets or four-dimensional time-lapse movies of rare events at rates that dramatically exceed human capabilities. From this, we quantify the effects of Taxol dose on spindle structure and kinetochore dynamics in dividing cells and of antigen strength on cytotoxic T lymphocyte engagement and lytic granule polarization at the immune synapse. Overall, smartLLSM efficiently detects rare events within heterogeneous cell populations and records these processes with high spatiotemporal four-dimensional imaging over statistically significant replicates. smartLLSM uses artificial intelligence-based instrument control to switch between epiflouorescence and lattice light-sheet microscopy to monitor cells at the population level while also capturing multicolor three-dimensional datasets of rare events of interest.

Genetically encoded voltage indicators are emerging tools for monitoring voltage dynamics with cell-type specificity. However, current indicators enable a narrow range of applications due to poor performance under two-photon microscopy, a method of choice for deep-tissue recording. To improve indicators, we developed a multiparameter high-throughput platform to optimize voltage indicators for two-photon microscopy. Using this system, we identified JEDI-2P, an indicator that is faster, brighter, and more sensitive and photostable than its predecessors. We demonstrate that JEDI-2P can report light-evoked responses in axonal termini of Drosophila interneurons and the dendrites and somata of amacrine cells of isolated mouse retina. JEDI-2P can also optically record the voltage dynamics of individual cortical neurons in awake behaving mice for more than 30 min using both resonant-scanning and ULoVE random-access microscopy. Finally, ULoVE recording of JEDI-2P can robustly detect spikes at depths exceeding 400 μm and report voltage correlations in pairs of neurons.

In vivo two-photon calcium imaging (2PCI) is a technique used for recording neuronal activity in the intact brain. It is based on the principle that, when neurons fire action potentials, intracellular calcium levels rise, which can be detected using fluorescent molecules that bind to calcium. This Primer is designed for scientists who are considering embarking on experiments with 2PCI. We provide the reader with a background on the basic concepts behind calcium imaging and on the reasons why 2PCI is an increasingly powerful and versatile technique in neuroscience. The Primer explains the different steps involved in experiments with 2PCI, provides examples of what ideal preparations should look like and explains how data are analysed. We also discuss some of the current limitations of the technique, and the types of solutions to circumvent them. Finally, we conclude by anticipating what the future of 2PCI might look like, emphasizing some of the analysis pipelines that are being developed and international efforts for data sharing.

In many areas of the brain, neural populations act as a coordinated network whose state is tied to behavior on a millisecond timescale. Two-photon (2p) calcium imaging is a powerful tool to probe such network-scale phenomena. However, estimating the network state and dynamics from 2p measurements has proven challenging because of noise, inherent nonlinearities and limitations on temporal resolution. Here we describe Recurrent Autoencoder for Discovering Imaged Calcium Latents (RADICaL), a deep learning method to overcome these limitations at the population level. RADICaL extends methods that exploit dynamics in spiking activity for application to deconvolved calcium signals, whose statistics and temporal dynamics are quite distinct from electrophysiologically recorded spikes. It incorporates a new network training strategy that capitalizes on the timing of 2p sampling to recover network dynamics with high temporal precision. In synthetic tests, RADICaL infers the network state more accurately than previous methods, particularly for high-frequency components. In 2p recordings from sensorimotor areas in mice performing a forelimb reach task, RADICaL infers network state with close correspondence to single-trial variations in behavior and maintains high-quality inference even when neuronal populations are substantially reduced. Zhu et al. develop a deep learning method to precisely infer single-trial neural dynamics from calcium imaging with subframe temporal resolution, which shows improvement over the state-of-the-art methods in capturing high-frequency dynamics and predicting behavior.

Modern neural interfaces allow access to the activity of up to a million neurons within brain circuits. However, bandwidth limits often create a trade-off between greater spatial sampling (more channels or pixels) and the temporal frequency of sampling. Here we demonstrate that it is possible to obtain spatio-temporal super-resolution in neuronal time series by exploiting relationships among neurons, embedded in latent low-dimensional population dynamics. Our novel neural network training strategy, selective backpropagation through time (SBTT), enables learning of deep generative models of latent dynamics from data in which the set of observed variables changes at each time step. The resulting models are able to infer activity for missing samples by combining observations with learned latent dynamics. We test SBTT applied to sequential autoencoders and demonstrate more efficient and higher-fidelity characterization of neural population dynamics in electrophysiological and calcium imaging data. In electrophysiology, SBTT enables accurate inference of neuronal population dynamics with lower interface bandwidths, providing an avenue to significant power savings for implanted neuroelectronic interfaces. In applications to two-photon calcium imaging, SBTT accurately uncovers high-frequency temporal structure underlying neural population activity, substantially outperforming the current state-of-the-art. Finally, we demonstrate that performance could be further improved by using limited, highbandwidth sampling to pretrain dynamics models, and then using SBTT to adapt these models for sparsely-sampled data.

Integration of multi-frequency sounds into a unified perceptual object is critical for recognizing syllables in speech. This “feature binding” relies on the precise synchrony of each component’s onset timing, but little is known regarding its neural correlates. We find that multi-frequency sounds prevalent in vocalizations, specifically harmonics, preferentially activate the mouse secondary auditory cortex (A2), whose response deteriorates with shifts in component onset timings. The temporal window for harmonics integration in A2 was broadened by inactivation of somatostatin-expressing interneurons (SOM cells), but not parvalbumin-expressing interneurons (PV cells). Importantly, A2 has functionally connected subnetworks of neurons preferentially encoding harmonic over inharmonic sounds. These subnetworks are stable across days and exist prior to experimental harmonics exposure, suggesting their formation during development. Furthermore, A2 inactivation impairs performance in a discrimination task for coincident harmonics. Together, we propose A2 as a locus for multi-frequency integration, which may form the circuit basis for vocal processing.

Voltage imaging enables monitoring neural activity at sub-millisecond and sub-cellular scale, unlocking the study of subthreshold activity, synchrony, and network dynamics with unprecedented spatio-temporal resolution. However, high data rates (>800MB/s) and low signal-to-noise ratios create bottlenecks for analyzing such datasets. Here we present VolPy, an automated and scalable pipeline to pre-process voltage imaging datasets. VolPy features motion correction, memory mapping, automated segmentation, denoising and spike extraction, all built on a highly parallelizable, modular, and extensible framework optimized for memory and speed. To aid automated segmentation, we introduce a corpus of 24 manually annotated datasets from different preparations, brain areas and voltage indicators. We benchmark VolPy against ground truth segmentation, simulations and electrophysiology recordings, and we compare its performance with existing algorithms in detecting spikes. Our results indicate that VolPy's performance in spike extraction and scalability are state-of-the-art.

Optical microscopy methods such as calcium and voltage imaging enable fast activity readout of large neuronal populations using light. However, the lack of corresponding advances in online algorithms has slowed progress in retrieving information about neural activity during or shortly after an experiment. This gap not only prevents the execution of real-time closed-loop experiments, but also hampers fast experiment–analysis–theory turnover for high-throughput imaging modalities. Reliable extraction of neural activity from fluorescence imaging frames at speeds compatible with indicator dynamics and imaging modalities poses a challenge. We therefore developed FIOLA, a framework for fluorescence imaging online analysis that extracts neuronal activity from calcium and voltage imaging movies at speeds one order of magnitude faster than state-of-the-art methods. FIOLA exploits algorithms optimized for parallel processing on GPUs and CPUs. We demonstrate reliable and scalable performance of FIOLA on both simulated and real calcium and voltage imaging datasets. Finally, we present an online experimental scenario to provide guidance in setting FIOLA parameters and to highlight the trade-offs of our approach. FIOLA is a pipeline for processing calcium or voltage imaging data. Its advantages include the fast speed and online processing.

Neuroscientists now routinely record the activity of large numbers of neurons at high temporal and spatial resolution. With these capabilities comes the promise of causally intervening during these recordings by perturbing neurons or changing experimental conditions, requiring tight integration between data acquisition, analysis, and manipulation. Unfortunately, solutions for real-time interventions are rare, difficult to design and implement, and remain largely unused. Here, we introduce improv , a software platform that allows users to flexibly specify and manage adaptive experiments to integrate data collection, preprocessing, visualization, and user-defined analytics. Using improv for streaming data analysis for two photon calcium imaging and behavior we demonstrate how access to online information can be used for automated, integrated experimentation. ### Competing Interest Statement The authors have declared no competing interest.

The broad accessibility of affordable and reliable recording equipment and its relative ease of use has enabled neuroscientists to record large amounts of neurophysiological and behavioral data. Given that most of this raw data is unlabeled, great effort is required to adapt it for behavioral phenotyping or signal extraction, for behavioral and neurophysiological data, respectively. Traditional methods for labeling datasets rely on human annotators which is a resource and time intensive process, which often produce data that that is prone to reproducibility errors. Here, we propose a deep learning-based image segmentation framework to automatically extract and label limb movements from movies capturing frontal and lateral views of head-fixed mice. The method decomposes the image into elemental regions (superpixels) with similar appearance and concordant dynamics and stacks them following their partial temporal trajectory. These 3D descriptors (referred as motion cues) are used to train a deep convolutional neural network (CNN). We use the features extracted at the last fully connected layer of the network for training a Long Short Term Memory (LSTM) network that introduces spatio-temporal coherence to the limb segmentation. We tested the pipeline in two video acquisition settings. In the first, the camera is installed on the right side of the mouse (lateral setting). In the second, the camera is installed facing the mouse directly (frontal setting). We also investigated the effect of the noise present in the videos and the amount of training data needed, and we found that reducing the number of training samples does not result in a drop of more than 5% in detection accuracy even when as little as 10% of the available data is used for training.

Big data problems frequently require processing datasets in a streaming fashion, either because all data are available at once but collectively are larger than available memory or because the data intrinsically arrive one data point at a time and must be processed online. Here, we introduce a computationally efficient version of similarity matching [1], a framework for online dimensionality reduction that incrementally estimates the top K-dimensional principal subspace of streamed data while keeping in memory only the last sample and the current iterate. To assess the performance of our approach, we construct and make public a test suite containing both a synthetic data generator and the infrastructure to test online dimensionality reduction algorithms on real datasets, as well as performant implementations of our algorithm and competing algorithms with similar aims. Among the algorithms considered we find our approach to be competitive, performing among the best on both synthetic and real data.

In vivo calcium imaging through microendoscopic lenses enables imaging of previously inaccessible neuronal populations deep within the brains of freely moving animals. However, it is computationally challenging to extract single-neuronal activity from microendoscopic data, because of the very large background fluctuations and high spatial overlaps intrinsic to this recording modality. Here, we describe a new constrained matrix factorization approach to accurately separate the background and then demix and denoise the neuronal signals of interest. We compared the proposed method against previous independent components analysis and constrained nonnegative matrix factorization approaches. On both simulated and experimental data recorded from mice, our method substantially improved the quality of extracted cellular signals and detected more well-isolated neural signals, especially in noisy data regimes. These advances can in turn significantly enhance the statistical power of downstream analyses, and ultimately improve scientific conclusions derived from microendoscopic data.

Cerebellar granule cells, which constitute half the brain's neurons, supply Purkinje cells with contextual information necessary for motor learning, but how they encode this information is unknown. Here we show, using two-photon microscopy to track neural activity over multiple days of cerebellum-dependent eyeblink conditioning in mice, that granule cell populations acquire a dense representation of the anticipatory eyelid movement. Initially, granule cells responded to neutral visual and somatosensory stimuli as well as periorbital airpuffs used for training. As learning progressed, two-thirds of monitored granule cells acquired a conditional response whose timing matched or preceded the learned eyelid movements. Granule cell activity covaried trial by trial to form a redundant code. Many granule cells were also active during movements of nearby body structures. Thus, a predictive signal about the upcoming movement is widely available at the input stage of the cerebellar cortex, as required by forward models of cerebellar control.

Optical imaging methods using calcium indicators are critical for monitoring the activity of large neuronal populations in vivo. Imaging experiments typically generate a large amount of data that needs to be processed to extract the activity of the imaged neuronal sources. While deriving such processing algorithms is an active area of research, most existing methods require the processing of large amounts of data at a time, rendering them vulnerable to the volume of the recorded data, and preventing real-time experimental interrogation. Here we introduce OnACID, an Online framework for the Analysis of streaming Calcium Imaging Data, including i) motion artifact correction, ii) neuronal source extraction, and iii) activity denoising and deconvolution. Our approach combines and extends previous work on online dictionary learning and calcium imaging data analysis, to deliver an automated pipeline that can discover and track the activity of hundreds of cells in real time, thereby enabling new types of closed-loop experiments. We apply our algorithm on two large scale experimental datasets, benchmark its performance on manually annotated data, and show that it outperforms a popular offline approach.

BACKGROUND:The preparation consisting of a head-fixed mouse on a spherical or cylindrical treadmill offers unique advantages in a variety of experimental contexts. Head fixation provides the mechanical stability necessary for optical and electrophysiological recordings and stimulation. Additionally, it can be combined with virtual environments such as T-mazes, enabling these types of recording during diverse behaviors. NEW METHOD:In this paper we present a low-cost, easy-to-build acquisition system, along with scalable computational methods to quantitatively measure behavior (locomotion and paws, whiskers, and tail motion patterns) in head-fixed mice locomoting on cylindrical or spherical treadmills. EXISTING METHODS:Several custom supervised and unsupervised methods have been developed for measuring behavior in mice. However, to date there is no low-cost, turn-key, general-purpose, and scalable system for acquiring and quantifying behavior in mice. RESULTS:We benchmark our algorithms against ground truth data generated either by manual labeling or by simpler methods of feature extraction. We demonstrate that our algorithms achieve good performance, both in supervised and unsupervised settings. CONCLUSIONS:We present a low-cost suite of tools for behavioral quantification, which serve as valuable complements to recording and stimulation technologies being developed for the head-fixed mouse preparation.

A major goal of the BRAIN Initiative is the development of technologies to monitor neuronal network activity during active information processing. Toward this goal, genetically encoded calcium indicator proteins have become widely used for reporting activity in preparations ranging from invertebrates to awake mammals. However, slow response times, the narrow sensitivity range of Ca2+ and in some cases, poor signal-to-noise ratio still limit their usefulness. Here, we review recent improvements in the field of neural activity-sensitive probe design with a focus on the GCaMP family of calcium indicator proteins. In this context, we present our newly developed Fast-GCaMPs, which have up to 4-fold accelerated off-responses compared with the next-fastest GCaMP, GCaMP6f. Fast-GCaMPs were designed by destabilizing the association of the hydrophobic pocket of calcium-bound calmodulin with the RS20 binding domain, an intramolecular interaction that protects the green fluorescent protein chromophore. Fast-GCaMP6f-RS06 and Fast-GCaMP6f-RS09 have rapid off-responses in stopped-flow fluorimetry, in neocortical brain slices, and in the intact cerebellum in vivo. Fast-GCaMP6f variants should be useful for tracking action potentials closely spaced in time, and for following neural activity in fast-changing compartments, such as axons and dendrites. Finally, we discuss strategies that may allow tracking of a wider range of neuronal firing rates and improve spike detection.