One of the main bottlenecks of blockchains is smart contract execution. To increase throughput, modern blockchains try to execute transactions in parallel. Unfortunately, however, common blockchain use cases introduce read-write conflicts between transactions, forcing sequentiality. We propose RapidLane, an extension for parallel execution engines that allows the engine to capture computations in conflicting parts of transactions and defer their execution until a later time, sometimes optimistically predicting execution results. This technique, coupled with support for a new construct for smart contract languages, allows one to turn certain sequential workloads into parallelizable ones. We integrated RapidLane into Block-STM, a state-of-the-art parallel execution engine used by several blockchains in production, and deployed it on the Aptos blockchain. Our evaluation shows that on commonly contended workloads, such as peer-to-peer transfers with a single fee payer and NFT minting, RapidLane yields up to 12× more throughput.

Advanced compiler technology is crucial for enabling machine learning applications to run on novel hardware, but traditional compilers fail to deliver performance, popular auto-tuners have long search times and expert-optimized libraries introduce unsustainable costs. To address this, we developed LoopTune, a deep reinforcement learning compiler that optimizes tensor computations in deep learning models for the CPU. LoopTune optimizes tensor traversal order while using the ultra-fast lightweight code generator LoopNest to perform hardware-specific optimizations. With a novel graph-based representation and action space, LoopTune speeds up LoopNest by 3.2x, generating an order of magnitude faster code than TVM, 2.8x faster than MetaSchedule, and 1.08x faster than AutoTVM, consistently performing at the level of the hand-tuned library Numpy. Moreover, LoopTune tunes code in order of seconds.

We assembled a semi-automated reconstruction of L2/3 mouse primary visual cortex from ∼250 × 140 × 90 μm3 of electron microscopic images, including pyramidal and non-pyramidal neurons, astrocytes, microglia, oligodendrocytes and precursors, pericytes, vasculature, nuclei, mitochondria, and synapses. Visual responses of a subset of pyramidal cells are included. The data are publicly available, along with tools for programmatic and three-dimensional interactive access. Brief vignettes illustrate the breadth of potential applications relating structure to function in cortical circuits and neuronal cell biology. Mitochondria and synapse organization are characterized as a function of path length from the soma. Pyramidal connectivity motif frequencies are predicted accurately using a configuration model of random graphs. Pyramidal cells receiving more connections from nearby cells exhibit stronger and more reliable visual responses. Sample code shows data access and analysis.

Learning from experience depends at least in part on changes in neuronal connections. We present the largest map of connectivity to date between cortical neurons of a defined type (layer 2/3 [L2/3] pyramidal cells in mouse primary visual cortex), which was enabled by automated analysis of serial section electron microscopy images with improved handling of image defects (250 × 140 × 90 μm3 volume). We used the map to identify constraints on the learning algorithms employed by the cortex. Previous cortical studies modeled a continuum of synapse sizes by a log-normal distribution. A continuum is consistent with most neural network models of learning, in which synaptic strength is a continuously graded analog variable. Here, we show that synapse size, when restricted to synapses between L2/3 pyramidal cells, is well modeled by the sum of a binary variable and an analog variable drawn from a log-normal distribution. Two synapses sharing the same presynaptic and postsynaptic cells are known to be correlated in size. We show that the binary variables of the two synapses are highly correlated, while the analog variables are not. Binary variation could be the outcome of a Hebbian or other synaptic plasticity rule depending on activity signals that are relatively uniform across neuronal arbors, while analog variation may be dominated by other influences such as spontaneous dynamical fluctuations. We discuss the implications for the longstanding hypothesis that activity-dependent plasticity switches synapses between bistable states.

We present LoopStack, a domain specific compiler stack for tensor operations, composed of a frontend, LoopTool, and an efficient optimizing code generator, LoopNest. This stack enables us to compile entire neural networks and generate code targeting the AVX2, AVX512, NEON, and NEONfp16 instruction sets while incorporating optimizations often missing from other machine learning compiler backends. We evaluate our stack on a collection of full neural networks and commonly used network blocks as well as individual operators, and show that LoopStack generates machine code that matches and frequently exceeds the performance of in state-of-the-art machine learning frameworks in both cases. We also show that for a large collection of schedules LoopNest's compilation is orders of magnitude faster than LLVM, while resulting in equal or improved run time performance. Additionally, LoopStack has a very small memory footprint - a binary size of 245KB, and under 30K lines of effective code makes it ideal for use on mobile and embedded devices.

Three-dimensional electron microscopy images of brain tissue and their dense segmentations are now petascale and growing. These volumes require the mass production of dense segmentation-derived neuron skeletons, multi-resolution meshes, image hierarchies (for both modalities) for visualization and analysis, and tools to manage the large amount of data. However, open tools for large-scale meshing, skeletonization, and data management have been missing. Igneous is a Python-based distributed computing framework that enables economical meshing, skeletonization, image hierarchy creation, and data management using cloud or cluster computing that has been proven to scale horizontally. We sketch Igneous's computing framework, show how to use it, and characterize its performance and data storage.

Neurons in the developing brain undergo extensive structural refinement as nascent circuits adopt their mature form. This physical transformation of neurons is facilitated by the engulfment and degradation of axonal branches and synapses by surrounding glial cells, including microglia and astrocytes. However, the small size of phagocytic organelles and the complex, highly ramified morphology of glia have made it difficult to define the contribution of these and other glial cell types to this crucial process. Here, we used large-scale, serial section transmission electron microscopy (TEM) with computational volume segmentation to reconstruct the complete 3D morphologies of distinct glial types in the mouse visual cortex, providing unprecedented resolution of their morphology and composition. Unexpectedly, we discovered that the fine processes of oligodendrocyte precursor cells (OPCs), a population of abundant, highly dynamic glial progenitors, frequently surrounded small branches of axons. Numerous phagosomes and phagolysosomes (PLs) containing fragments of axons and vesicular structures were present inside their processes, suggesting that OPCs engage in axon pruning. Single-nucleus RNA sequencing from the developing mouse cortex revealed that OPCs express key phagocytic genes at this stage, as well as neuronal transcripts, consistent with active axon engulfment. Although microglia are thought to be responsible for the majority of synaptic pruning and structural refinement, PLs were ten times more abundant in OPCs than in microglia at this stage, and these structures were markedly less abundant in newly generated oligodendrocytes, suggesting that OPCs contribute substantially to the refinement of neuronal circuits during cortical development.

Many approaches to 3D image segmentation are based on hierarchical clustering of supervoxels into image regions. Here we describe a distributed algorithm capable of handling a tremendous number of supervoxels. The algorithm works recursively, the regions are divided into chunks that are processed independently in parallel by multiple workers. At each round of the recursive procedure, the chunk size in all dimensions are doubled until a single chunk encompasses the entire image. The final result is provably independent of the chunking scheme, and the same as if the entire image were processed without division into chunks. This is nontrivial because a pair of adjacent regions is scored by some statistical property (e.g. mean or median) of the affinities at the interface, and the interface may extend over arbitrarily many chunks. The trick is to delay merge decisions for regions that touch chunk boundaries, and only complete them in a later round after the regions are fully contained within a chunk. We demonstrate the algorithm by clustering an affinity graph with over 1.5 trillion edges between 135 billion supervoxels derived from a 3D electron microscopic brain image.

Inhibitory neurons in mammalian cortex exhibit diverse physiological, morphological, molecular, and connectivity signatures. While considerable work has measured the average connectivity of several interneuron classes, there remains a fundamental lack of understanding of the connectivity distribution of distinct inhibitory cell types with synaptic resolution, how it relates to properties of target cells, and how it affects function. Here, we used large-scale electron microscopy and functional imaging to address these questions for chandelier cells in layer 2/3 of the mouse visual cortex. With dense reconstructions from electron microscopy, we mapped the complete chandelier input onto 153 pyramidal neurons. We found that synapse number is highly variable across the population and is correlated with several structural features of the target neuron. This variability in the number of axo-axonic ChC synapses is higher than the variability seen in perisomatic inhibition. Biophysical simulations show that the observed pattern of axo-axonic inhibition is particularly effective in controlling excitatory output when excitation and inhibition are co-active. Finally, we measured chandelier cell activity in awake animals using a cell-type-specific calcium imaging approach and saw highly correlated activity across chandelier cells. In the same experiments, in vivo chandelier population activity correlated with pupil dilation, a proxy for arousal. Together, these results suggest that chandelier cells provide a circuit-wide signal whose strength is adjusted relative to the properties of target neurons.

Article Figures and data Abstract Editor's evaluation Introduction Results Discussion Materials and methods Data availability References Decision letter Author response Article and author information Metrics Abstract Inhibitory neurons in mammalian cortex exhibit diverse physiological, morphological, molecular, and connectivity signatures. While considerable work has measured the average connectivity of several interneuron classes, there remains a fundamental lack of understanding of the connectivity distribution of distinct inhibitory cell types with synaptic resolution, how it relates to properties of target cells, and how it affects function. Here, we used large-scale electron microscopy and functional imaging to address these questions for chandelier cells in layer 2/3 of the mouse visual cortex. With dense reconstructions from electron microscopy, we mapped the complete chandelier input onto 153 pyramidal neurons. We found that synapse number is highly variable across the population and is correlated with several structural features of the target neuron. This variability in the number of axo-axonic ChC synapses is higher than the variability seen in perisomatic inhibition. Biophysical simulations show that the observed pattern of axo-axonic inhibition is particularly effective in controlling excitatory output when excitation and inhibition are co-active. Finally, we measured chandelier cell activity in awake animals using a cell-type-specific calcium imaging approach and saw highly correlated activity across chandelier cells. In the same experiments, in vivo chandelier population activity correlated with pupil dilation, a proxy for arousal. Together, these results suggest that chandelier cells provide a circuit-wide signal whose strength is adjusted relative to the properties of target neurons. Editor's evaluation This paper will be of high interest to a broad audience of neuroscientists as it provides a major advancement of our understanding of cortical circuits. The quality and quantitative nature of the neuroanatomical reconstructions at synaptic resolution are remarkable. Complementing the reconstructions with computational modeling and activity measurements, the study proposes a likely circuit function for a specific inhibitory cell type during behavior. https://doi.org/10.7554/eLife.73783.sa0 Decision letter eLife's review process Introduction The diversity of inhibitory cell types in mammalian neocortex, each with distinctive projection patterns and physiology, implies a rich role in cortical computation (Petilla Interneuron Nomenclature Group et al., 2008; Fino et al., 2013; Freund and Buzsáki, 1996; Jiang et al., 2015; Kepecs and Fishell, 2014; Kubota, 2014). To understand the role of a particular cell type in the brain, knowledge about the cell-type identity, connectivity within and between cell types and activity is required. For example, vasoactive intestinal polypeptide (VIP)-positive interneurons preferentially inhibit other inhibitory interneurons (Lee et al., 2013; Pfeffer et al., 2013; Pi et al., 2013) and are active during behaviors such as locomotion (Fu et al., 2014) and whisking (Lee et al., 2013; Muñoz et al., 2017). This established a key role for VIP neurons in the dynamic regulation of cortical inhibition under the control of brain-state-dependent modulators (Alitto and Dan, 2012; Kawaguchi, 1997; Lee et al., 2013; McGinley et al., 2015; Muñoz et al., 2017; Pakan et al., 2016; Pi et al., 2013; Polack et al., 2013; Reimer et al., 2014; Stryker, 2014; Vinck et al., 2015; Zhang et al., 2014). Similar efforts have elucidated circuit roles of other cell types, including parvalbumin (PV)-expressing basket cells (Atallah et al., 2012; Nienborg et al., 2013; Packer and Yuste, 2011; Wilson et al., 2012) and somatostatin (SST)-expressing Martinotti cells (Adesnik et al., 2012; Muñoz et al., 2017; Nienborg et al., 2013; Silberberg and Markram, 2007; Wang et al., 2004; Wilson et al., 2012). The chandelier cell (ChC) has properties that, at first glance, should make it a good candidate to be among the better-understood inhibitory cell types. Sometimes referred to as axo-axonal cells, ChCs are GABAergic interneurons-characterized vertical axonal 'candles' or 'cartridges' (Jones, 1975; Peters et al., 1982; Szentágothai and Arbib, 1974) that synapse almost exclusively with the axon initial segment (AIS) of excitatory pyramidal neurons (PyCs) (DeFelipe et al., 1985; Fairén and Valverde, 1980; Somogyi, 1977; Somogyi et al., 1982). This pattern of connectivity suggests a unique role for ChCs as the AIS is a specialized compartment whose unique ion channel distribution makes it the principal site of action potential generation (Kole and Stuart, 2012; Kole et al., 2007; Palmer and Stuart, 2006). Each ChC connects to 30–50% of PyCs within a 200-µm-wide axonal field (Wang et al., 2019), resulting in a small number of ChCs innervating a large number of PyCs in L2/3 (Inan et al., 2013). It follows that ChCs are particularly well positioned to exert powerful control over PyCs in superficial layers and, as a result, the entire cortical activity. Despite their highly specific connectivity, how ChCs affect neuronal circuits remains enigmatic (Inan and Anderson, 2014; Woodruff et al., 2010). One reason is the variability in the strength of ChC targeting. PyC populations in different brain regions and layers are contacted by diverse numbers of AIS-targeting boutons (DeFelipe et al., 1985; Veres et al., 2014; Wang and Sun, 2012). Consistent with this observation, activation of the ChC population in vivo has diverse effects on nearby PyCs, ranging from strong inhibition to a lack of response (Lu et al., 2017) despite apparently dense ChC connectivity (Inan et al., 2013). The logic underlying this variability, and thus the heterogeneity of ChC influence on the downstream targets, is largely unknown, although there are indications that PyCs with different long-range projection targets can have different magnitudes of ChC inhibition (Fariñas and DeFelipe, 1991b; Lu et al., 2017). A second reason why the role of ChCs has remained enigmatic is functional. While several individual sources of synaptic input into ChCs have been identified (Jiang et al., 2015; Lu et al., 2017), what drives ChC activity and how that, in turn, manifests itself into circuit-wide dynamics in a behaving animal remain unknown. To improve our understanding of both the circuit organization and function of ChCs, we took a multipronged approach. We used large-scale serial-section electron microscopy (EM) (Bock et al., 2011; Kasthuri et al., 2015; Lee et al., 2016) to map synaptic input onto the AIS across L2/3 PyCs in a volume of mouse primary visual cortex (Video 1). By using automated dense segmentation and synapse detection, we obtained a reconstruction of ChC axons and PyCs in a volume of approximately 3.6 × 106 µm3. The resolution and completeness afforded by this approach allowed us to infer underlying principles governing not only the presence but also key properties of ChC connectivity. In particular, it allowed us to study the variability of the synaptic connectivity onto different pyramidal cells and compare it with perisomatic inhibition, which is formed by different cell types. To address how ChC inhibition influences PyC activity at the cellular level, we use biophysical simulations and map input-output relationships that show the unique impact of axo-axonic over other forms of inhibition. Finally, we use a novel genetic approach to selectively measure ChC activity in the visual cortex of awake behaving mice being presented with visual stimuli. Video 1 Download asset This video cannot be played in place because your browser does support HTML5 video. You may still download the video for offline viewing. Download as MPEG-4 Download as WebM Download as Ogg A rendering of the electron microscopy (EM) reconstructions from this dataset demonstrating the mapping of chandelier cell (ChC) inputs onto layer 2/3 pyramidal neurons (PyCs). Video begins with four gray PyCs with only their somatic regions and axon initial segment (AIS) region shown. An individual pink ChC fragment is slowly revealed over time as the reconstruction is followed along to all the locations that it synapses onto. Note that the portions of that axon that are far from the four PyCs are excluded from the rendering for clarity. Then a second, purple axon fragment is revealed in the same fashion. Third, all the ChC fragments that synapse onto these four PyCs are revealed simultaneously, each with their unique color. Finally, the scene reveals all the PyCs in this structural dataset, and all the ChC branches reconstructed in the dataset are revealed in red. Results A densely segmented EM volume of layer 2/3 primary visual cortex EM offers the ability to identify every synapse and trace every neurite in a volume of tissue. In order to make precise measurements of the structure of cortical circuits, we prepared, sectioned, and imaged an EM volume of layer 2/3 (L2/3) from primary visual cortex (Figure 1A) of a P36 male mouse, spanning approximately 250 µm × 140 µm × 90 µm, with 40-nm-thick sections imaged at 3.58 × 3.58 nm/pixel with transmission EM (Figure 1B; see Materials and methods for details). Performing EM for such a large cortical volume is necessary in order to measure connectivity at a sufficient degree of precision. Specifically, we needed to be able to follow a multitude of individual neuronal processes for large distances throughout the volume and comprehensively identify synapses. Such circuit reconstruction at scale also requires intensive computational processing (Berning et al., 2015; Dorkenwald et al., 2017; Jain et al., 2010; Januszewski et al., 2018). We used a series of novel machine learning-based methods to perform image alignment, automated segmentation, and synapse detection for the volume (Figure 1C; Turner et al., 2020). Nonetheless, proofreading remains necessary for precise measurements of anatomy and connectivity. The initial segmentation identified small supervoxels that were agglomerated into cells, and we built a novel cloud-based proofreading system to edit the agglomerations and perform targeted error correction (Dorkenwald et al., 2019; Dorkenwald et al., 2020a) (see Materials and methods). The same dataset has also been used concurrently in other studies (Buchanan et al., 2021; Dorkenwald et al., 2019; Turner et al., 2020). We conclude that we established a high-throughput EM pipeline that, combined with novel image-processing computational tools, allows the ability to reconstruct entire cortical circuits at unprecedented spatial detail. Figure 1 with 4 supplements see all Download asset Open asset A map of axon initial segment (AIS) input from electron microscopy (EM). (A) A block of tissue was selected from L2/3 of mouse V1 and processed in an EM pipeline. (B) Serial 40 nm sections were imaged computationally aligned into a volume. (C) Image annotation pipeline. Left: images were taken with 3.58 × 3.58 nm pixels. Scale bar is 500 nm. Center: the neuropil was densely segmented and targeted proofreading was done to correct pyramidal neurons (PyCs) and other objects of interest. Right: automated synapse detection identified pre- and postsynaptic locations for synapses and was followed by targeted proofreading for false positives. To interactively view the dataset in 3D, visit https://www.microns-explorer.org/chc/soma/all_by_type. (D) For each PyC with significant AIS in the volume, we started with the overall morphology and synaptic inputs (cyan dots) and computationally extracted the AIS (dark gray) and its synaptic inputs (cyan arrows). All AIS bounds can be seen in 3D at https://www.microns-explorer.org/chc/soma/ais_bounds. (E) Soma, AIS, and AIS synaptic inputs for all PyC analyzed. The volume is rotated so that the average AIS direction is exactly downward. Note that dendrites and higher-order axon branches are omitted for clarity. (F) Histogram of synapses per AIS. Synapse data can be found in Supplementary file 2. A complete map of synaptic input to the AISs of an excitatory network We manually identified all cell bodies in the volume (n = 547) and manually classified each as excitatory PyCs (n = 416), inhibitory (n = 34), or glia (n = 97) based on morphology and ultrastructural features such as dendritic spines. Dendritic and axonal arbors of all neurons with cell bodies were proofread to correct segmentation errors. The full dataset, including soma classifications, can be explored online at https://microns-explorer.org/chc/soma/all_by_type. We next mapped all synaptic input onto the AIS of excitatory cells that had a complete AIS in the volume. Since we could not robustly identify the molecular components of the AIS from EM imagery, we opted instead for a purely structural definition: from the axon hillock (whether it emerged from the soma or a proximal dendrite) to the most proximal of the first branch point, beginning of myelination, or the volume exit. Only cells with at least 40 µm of AIS within the volume were considered, a distance found to contain almost all AIS synapses on more complete reconstructions (see also Figure 2I). For each PyC with a complete AIS (N = 153), we manually marked points as the top and bottom of the AIS and used them to computationally specify the AIS and its synaptic input (Figure 1D and E, Figure 1—figure supplement 1, https://microns-explorer.org/chc/soma/ais_bounds). All synapses were proofread to remove false positives from the automated synapse detection. This resulted in a total of 1929 AIS synapses across 153 PyCs, for a mean of 12.6 synapses per AIS (for examples, see Figure 1—figure supplements 2–3 and Supplementary file 1). Importantly, we observed a remarkable diversity of total inputs, ranging from 1 to 32 synapses per AIS (Figure 1F), pointing to differing magnitudes of AIS innervation across the PyC population. Figure 2 with 4 supplements see all Download asset Open asset Characterization of axon initial segment (AIS) input. (A) Classification and proofreading workflow. Across AIS synaptic inputs, morphology and connectivity were used to distinguish chandelier cell (ChC) from non-ChC axons. ChCs were given full proofreading to get as-complete-as-possible arbors. For non-ChC inputs, all AIS-targeting boutons were proofread to ensure they were not due to merged-in ChC axons. The pipeline was repeated until no new axons were identified. (B) Axons that exclusively synapsed onto pyramidal neuron (PyC) AISes were classified as ChCs. Example below shows an axon targeting the AIS of several PyCs (red arrowheads). (C) Axons that showed mixed AIS and non-AIS targeting were classified as non-ChC. Example at left shows an axon (black) targeting an AIS (red arrowheads) and a nearby soma (black arrowheads). Example at right shows an axon (black) targeting a dendrite (black arrowheads) and an AIS (red arrowheads). (E) All ChC objects identified from the AIS survey. (F) Single PyC soma and AIS shown with presynaptic axons. Reds indicate ChCs, purples indicate non-ChCs. Axons are truncated to the region near AIS contacts for clarity. For the full data in 3D, please visit https://www.microns-explorer.org/chc/ais/80. (G) Example ChC synapse imagery for the AIS in (F). The AIS outlined in blue, ChC bouton in red. Image panels are 1 µm × 1 µm. (H) Example non-ChC synapse imagery onto the AIS in (F). As in (G) but with purple outlines for boutons. (I) Histogram of distance from AIS base for ChC (red) and non-ChC (gray) synapses. (J) Distribution of AIS synapses per connection for ChCs. Inset: semilog-y synapses per connection for all ChCs (red dots) and only ChC axon fragments with more than 20 synapses (black dots), with Poisson fit to the complete data (dashed line). (K) Distribution of AIS synapses per connection for non-ChCs. (L) Number of distinct presynaptic ChC axons per AIS. (M) Distribution of mean number of synapses in a ChC connection for each AIS. (N) Distribution of the number of synapses in the most numerous ChC connection on each AIS. Synapse data can be found in Supplementary file 2. PyC AIS input is a mix of chandelier and non-chandelier synapses While ChCs are the only cell type to specifically target the AIS, other cell types can also form synapses on the AIS (Gonchar et al., 2002; Gour et al., 2021; Kisvárday et al., 1985; Somogyi, 1977). To identify which AIS input synapses came from ChCs and which did not (Figure 2A), we examined the morphology and ultrastructure of every axon presynaptic to the AIS of any of the PyCs (Figure 1—figure supplements 2–3, Supplementary file 1). By following all axons forming synapses onto AIS of the 153 PyCs considered in the cortical volume, we found axons that exclusively target the AIS of PyCs (Figure 2B, Figure 2—figure supplement 1) and axons that targeted a mixture of compartments (Figure 2C). Axons exclusively targeting AIS were manually proofread and re-evaluated, while mixed-target cells were proofread to discover any falsely merged AIS-targeting fragments. AIS-exclusive targeting axons were classified as ChCs, while mixed-target axons were classified as non-ChCs (see Materials and methods). In total, we found 1127 AIS synapses from ChCs (122 axon fragments and 2 ChCs with soma in the volume, see Figure 2—figure supplement 2) and 802 AIS synapses from non-ChC axon fragments (Figure 2D and E). Each AIS and its complete presynaptic input can be viewed online (e.g., https://www.microns-explorer.org/chc/ais/1 — see Supplementary file 1 for all URLs). We conclude that while the majority of AIS synapses in our volume originate from ChC axons, more than 40% originate from non-ChC axons. To understand how ChC and non-ChC axons differ in how they target the AIS, we looked at the precise location of their synapses. We found that ChC synapses onto PyCs were predominantly located on the AIS in a region between 10 and 40 µm from the axon hillock, consistent with previous observations (Veres et al., 2014), while non-ChC synapses were most common near the soma, but widely distributed across the AIS (Figure 2F). Indeed, ChC and non-ChC synapses were often intermingled within individual AISs (Figure 2G–I). Even within the limited span of the AIS, ChC axons were not distributed randomly. From the 3D reconstructions of ChC boutons targeting a single AIS, we noticed that instead of being scattered on the surface of the AIS (as observed for somatic inhibitory input), multiple ChC boutons from different axons were clustered at distinct points on the AIS (Figure 2—figure supplement 3). Under such ChC clusters, we typically observed a cisternal organelle (CO), an AIS-specific endoplasmic reticulum specialization associated with a complex assortment of molecular components (King et al., 2014) that actively contribute to the local modulation of calcium (Lipkin et al., 2021). Consistent with previous qualitative observations (Benedeczky et al., 1994; King et al., 2014), we found that ChC synapses were significantly closer to COs than expected by spatially shuffling synapse locations within and between AISs (Figure 2—figure supplement 4). Taken together, our results show that not only do ChCs target a more specific region on the AIS than non-ChCs, but that ChC boutons frequently form clustered synapses with other ChCs at the site of a specialized postsynaptic organelle. Distribution of AIS input by cell type An AIS can receive input from many different ChCs, each with potentially multiple synaptic contacts. We use 'synapse' to refer to a single anatomical synapse and 'connection' to indicate the collection of synapses between a given presynaptic axon branch and postsynaptic AIS, comprising one or more synapses. The ability to discriminate each presynaptic axon in our data offers the opportunity to look at the complete map of synaptic input and how it is organized by the identity of the presynaptic axon. It is likely that multiple axon branches belong to the same few ChCs (either the two found in the volume or others outside), but this identity cannot be reconstructed within the volume. However, based on whole-cell reconstructions showing that different branches of ChC axons rarely converge onto the same cartridge (Blazquez-Llorca et al., 2015; Gouwens et al., 2019), it is also likely that most distinct ChC axon branches targeting the same AIS come from distinct cells. However, such convergence cannot be ruled out. Individual ChC connections had between 1 and 9 synapses, and their distribution was well-fit by a geometric distribution (exponent: 0.44, Figure 2J). Notably, the synapse count included both standard multisynapse connections that characterize ChC cartridges, as well as numerous single-synapse connections. This distribution of connections was observed even when we only considered larger axon fragments (more than 20 synapses) (Figure 2J, inset), suggesting that it is not due only to highly fragmented reconstructions. In contrast, non-ChC connections comprised only a single synapse in 94% of examples (Figure 2K) and all non-ChC connections had three or fewer synapses in total. Our data suggest that while all high synapse-count AIS connections originate from ChC axons, both ChC and non-ChC form numerous weaker, often single-synapse, connections. For a given AIS, the impact of ChC input depends not on a single connection, but on the combination of synapses from potentially multiple presynaptic ChCs (Inan et al., 2013). We found that individual PyCs received input from between 0 and 9 distinct ChC axons (mean 3.3 ChC axons per AIS) and those AIS that received ChC input had a mean of 2.3 synapses per connection (Figure 2L and M) and almost 80% of AISes (122/153) still had at least one cartridge with two or more synapse (Figure 2N). Thus, despite the frequency of weak ChC connections, due to convergent input from many cells most PyCs nonetheless have at least one typical ChC cartridge. ChC connectivity onto pyramidal cells is highly variable and correlates with properties of target cells We next asked how ChC connectivity was related to the individual properties of the L2/3 PyCs targeted. Because different PyCs had different amounts of axonal arbor in the volume, we restricted our analysis to a consistent initial region that would both cover the typical molecularly defined AIS and include as many individual cells as possible. Based on the distribution of ChC synapses (Figure 2F), we used the first 37 µm of structural AIS for all cells, a span that contained 97% of ChC synapses (Figure 3A) while omitting only one PyC due to an insufficient length of AIS within the volume. Figure 3 with 3 supplements see all Download asset Open asset Structural properties associated with chandelier cell (ChC) synaptic input. (A) For each axon initial segment (AIS), we look only at synapses on the first 37 µm, which captures 97% of all ChC synapses while omitting as few cells as possible from analysis. (B) Soma location for cells with complete soma in the volume, colored by total ChC synapse count. Pia direction is up. (C) Distribution of total ChC synapse count across pyramidal neurons (PyCs). (D) Distribution of total somatic synapses across PyCs. (E) ChC synapse count vs. six structural properties of soma and AISes: depth, AIS radius, non-ChC AIS synapse count, soma synapse density, soma area, and soma synapse count. (F) Pearson correlation matrix between all structural properties. Entries with an asterisk are significant (p<0.05) after Holm–Sidak multiple test correction. (G) Independent components analysis (ICA) components for PyC structural properties. Components are oriented so that the highest loading element is positive. Each component is labeled with an approximate interpretation of its combination of properties. (H) Standardized ordinary least-squares (OLS) regression coefficients for ChC vs. the three ICA components. Bars indicate 95% confidence interval; stars indicate significance after Holm–Sidak multiple test correction. ***p<0.001. (I) Scatterplot and linear fit of residual ChC synapse count vs. depth after fitting the other two components. Shaded region indicates the 95% confidence interval estimated from bootstrap (N = 1000). (J) Same as (I), but for the soma inhibition component. (K) Same as (I), but for soma size component. Synapse and AIS data can be found in Supplementary file 2. ChC synapses were found on 95% of PyCs (144/152), but there was striking variability in the total number of ChC synapses (mean: 7.4 ± 5.3 synapses), with individual PyCs receiving between 0 and 25 ChC synapses (Figure 3B and C). Effectively, some L2/3 PyCs escape ChC input entirely while others are strongly innervated by ChCs. We asked if this variability was reflected in nearby perisomatic input, which for PyCs is thought to be almost entirely inhibitory (Davis and Sterling, 1979; Fariñas and DeFelipe, 1991a; White and Rock, 1980; Wildenberg et al., 2021). We confirmed this in our data; 81/81 randomly sampled somatic synapses on five PyCs were from an inhibitory axon, as were all somatic synapses (N = 75) on an additional randomly selected PyC (see Materials and methods). Using similar methods as for AIS synapses, we identified all somatic synapses for the 120/152 PyC whose cell bodies were fully contained in the volume (Figure 3—figure supplement 1). All PyCs had numerous somatic input synapses (47–113 synapses, Figure 3D). Notably, among those cells with fully measured somatic synapses, the total number of ChC synapses had a coefficient of variation (CV) of 0.73, compared to a CV of 0.18 for the number of perisomatic PyC synapses. We conclude that axo-axonic ChC synapse counts exhibits substantially higher variability than perisomatic synapse counts for PyCs. To explore the logic of the observed heterogeneity, we asked if the total number of ChC synapses that a PyC receives was associated with other structural properties of the target cell. For consistency in our measurements, we focused on the soma and the AIS. Specifically, for each PyC with a complete soma, we measured its depth within L2/3, mean AIS radius, number of non-ChC AIS synapses, number of synaptic inputs onto the soma, soma surface area, and soma synapse density. Strikingly, we found statistically significant correlations between the number of ChC synapses and each property except for non-ChC AIS synapses (Figure 3E and F). However, we also found that various size and synaptic input properties of each PyC were significantly correlated among themselves (Figure 3F). To disentangle these correlations, we performed independent components analysis (ICA), a variant of principal components analysis that yields fully uncorrelated components (Comon, 1994). ICA attributed the observed correlations to three main postsynaptic PyC components: the cortical depth of the soma (i.e., location), the amount of somatic inhibition (i.e., soma synapses, soma synapse density, and non-ChC AIS synapses), and the perisomatic PyC size (i.e., soma area and AIS radius) (Figure 3G). We next asked if these components were associated with differences in total number of ChC synapses. We performed multivariate ordinary least-squares (OLS) regression on the number of ChC synapses against the three ICA components for each PyC. All three components showed significant correlation with the number of ChC synapses, with the three together explaining 45% of the variance in ChC synapse number (Figure 3H–K). We report coefficients from z-scored variables. First, deeper PyCs received fewer ChC synapses (coefficient: –0.47). This is consistent with the observation that some ChC axons have denser axonal arbors in upper L2/3 (Wang et al., 2019) and our observation of a higher absolute number of ChC synapses in the upper part of our volume (Figure 3—figure supplement 2). Second, perisomatic inhibition was positively correlated with the number of ChC synapses (coefficient: 0.33) — that is, PyCs with more ChC synapses also receive more synapses from other, non-ChCs at their soma. Third, larger cells received more ChC synapses (coefficient: 0.34). Taken together, our data support that the amount of ChC synapses onto PyCs is influenced by the shape, location, and inhibitory connectivity of each target cell. Properties of individual target cells influence the number of chandelier synapses on the AIS The same number of ChC synapses onto a single PyC could result from different combinations of the number of connections and the number of synapses per connection. (Figure 4A). Variability in the total number of ChC synapses along PyCs could thus be attributed to either or both connectivity properties. To understand the distinct role of both of these connectivity properties separately, we measured both the number of connections (i.e., unique presynaptic ChC axon branches) and the number of synapses per connection for all PyCs. We found that both the number of connections and mean synapses per connection ranged widely (1–9 connections and 1–7 synapses per connection), but were uncorrelated (r = −0.08, p<0.36, Figure 4B). Moreover, using the same perisomatic components OLS approach described above, we found no significant relationship between synapses per connection and soma depth or inhibition, although there was a modest positive correlation with soma size (Figure 4C and D). In contrast, we found that the number of connections was related to all three PyC properties in a similar pattern as total ChC synapses (Figure 4E and F). This suggests that the number of distinct ChC axon branches contacting a cell and the number of synapses per connection are regulated by different processes. Figure 4 with 1 su

The activity and connectivity of inhibitory cells has a profound impact on the operation of neuronal networks. While the average connectivity of many inhibitory cell types has been characterized, we still lack an understanding of how individual interneurons distribute their synapses onto their targets and how heterogeneous the inhibition is onto different individual excitatory neurons. Here, we use large-scale volumetric electron microscopy (EM) and functional imaging to address this question for chandelier cells in layer 2/3 of mouse visual cortex. Using dense morphological reconstructions from EM, we mapped the complete chandelier input onto 153 pyramidal neurons. We find that the number of input synapses is highly variable across the population, but the variability is correlated with structural features of the target neuron: soma depth, soma size, and the number of perisomatic synapses received. Functionally, we found that chandelier cell activity in vivo was highly correlated and tracks pupil diameter, a proxy for arousal state. We propose that chandelier cells provide a global signal whose strength is individually adjusted for each target neuron. This approach, combining comprehensive structural analysis with functional recordings of identified cell types, will be a powerful tool to uncover the wiring rules across the diversity of cortical cell types.

These are data tables released as part of the MICrONS project, and contain annotations on the EM data and reconstructions. You can find more information about the dataset and the project at microns-explorer.org.

Summary We present a semi-automated reconstruction of L2/3 mouse primary visual cortex from 3 million cubic microns of electron microscopic images, including pyramidal and inhibitory neurons, astrocytes, microglia, oligodendrocytes and precursors, pericytes, vasculature, mitochondria, and synapses. Visual responses of a subset of pyramidal cells are included. The data are being made publicly available, along with tools for programmatic and 3D interactive access. The density of synaptic inputs onto inhibitory neurons varies across cell classes and compartments. We uncover a compartment-specific correlation between mitochondrial coverage and synapse density. Frequencies of connectivity motifs in the graph of pyramidal cells are predicted quite accurately from node degrees using the configuration model of random graphs. Cells receiving more connections from nearby cells exhibit stronger and more reliable visual responses. These example findings illustrate the resource’s utility for relating structure and function of cortical circuits as well as for neuronal cell biology.

Convolutional nets have been shown to achieve state-of-the-art accuracy in many biomedical image analysis tasks. Many tasks within biomedical analysis domain involve analyzing volumetric (3D) data acquired by CT, MRI and Microscopy acquisition methods. To deploy convolutional nets in practical working systems, it is important to solve the efficient inference problem. Namely, one should be able to apply an already-trained convolutional network to many large images using limited computational resources. In this paper we present PZnet, a CPU-only engine that can be used to perform inference for a variety of 3D convolutional net architectures. PZNet outperforms MKL-based CPU implementations of PyTorch and Tensorflow by more than 3.5x for the popular U-net architecture. Moreover, for 3D convolutions with low featuremap numbers, cloud CPU inference with PZnet outperforms cloud GPU inference in terms of cost efficiency.

Fast convolutions via transforms, either Winograd or FFT, had emerged as a preferred way of performing the computation of convolutional layers, as it greatly reduces the number of required operations. Recent work shows that, for many layer structures, a well--designed implementation of fast convolutions can greatly utilize modern CPUs, significantly reducing the compute time. However, the generous amount of shared L3 cache present on modern CPUs is often neglected, and the algorithms are optimized solely for the private L2 cache. In this paper we propose an efficient `L3 Fusion` algorithm that is specifically designed for CPUs with significant amount of shared L3 cache. Using the hierarchical roofline model, we show that in many cases, especially for layers with fewer channels, the `L3 fused` approach can greatly outperform standard 3 stage one provided by big vendors such as Intel. We validate our theoretical findings, by benchmarking our `L3 fused` implementation against publicly available state of the art.

Winograd-based convolution has quickly gained traction as a preferred approach to implement convolutional neural networks (ConvNet) on various hardware platforms because it could require fewer floating point operations than FFT-based or direct convolutions. In this paper, we analyze the theoretical performances of three methods (regular FFT-, Gauss-FFT-, and Winograd-based convolutions), as well as compare their highly optimized implementations on modern multi- and many-core CPUs. With all three implementations employing the same optimizations on modern CPUs, our experimental results with modern ConvNets show that the FFT-based implementations generally outperform the Winograd-based approach, which is contrary to the popular belief. To understand the results, we use a Roofline performance model to analyze the three implementations in detail, by looking at each of their computation phases and by considering not only the number of floating point operations, but also the memory bandwidth and the cache sizes. The performance analysis explains why, and under what conditions, the FFT-based implementations outperform the Winograd-based one, on modern CPUs.

Winograd-based convolution has quickly gained traction as a preferred approach to implement convolutional neural networks (ConvNet) on various hardware platforms because it requires fewer floating point operations than FFT-based or direct convolutions. This paper compares three highly optimized implementations (regular FFT--, Gauss--FFT--, and Winograd--based convolutions) on modern multi-- and many--core CPUs. Although all three implementations employed the same optimizations for modern CPUs, our experimental results with two popular ConvNets (VGG and AlexNet) show that the FFT--based implementations generally outperform the Winograd--based approach, contrary to the popular belief. To understand the results, we use a Roofline performance model to analyze the three implementations in detail, by looking at each of their computation phases and by considering not only the number of floating point operations, but also the memory bandwidth and the cache sizes. The performance analysis explains why, and under what conditions, the FFT--based implementations outperform the Winograd--based one, on modern CPUs.

Since convolutional layers are computationally expensive and dominate the total execution time of modern deep ConvNets [13, 16, 18, 19], many efforts have been made to improve the performance of the convolutional primitives for CPUs [1, 7, 20, 25, 27], GPUs [4, 8, 15, 21] or both [26]. Initially, several approaches using FFT–based convolutions were proposed [15, 21, 25, 26]. Recent work by Lavin et al. on Winograd–based convolutions [14] demonstrated a great speedup, which shifted the focus from FFT–based to Winograd–based implementations, as it became widely accepted that the Winograd–based approach provides greater reduction in the number of operations required by the algorithm, especially for small kernels (e.g. 3 × 3). A well optimized manycore CPU implementation [3, 12] of the Winograd approach can improve the performances by more than 3X. The main reduction in operations in the Winograd method, compared to FFT, comes from the fact that it works with real numbers. However, due to its numerical instability, the Winograd method can only use small tile (transform) sizes [7, 14, 22], which result in a larger amount of required data movement to and from memory. In contrast, the FFT–based method does not suffer from such instability, thus larger tile sizes can be used, which can partially reduce the number of required operations and greatly reduce the amount of data movements; these savings can, in certain cases, offset the increase in the number of operations due to complex arithmetic. These observations raise the question, under what conditions the Winograd-based approach performs better than the FFT– based approach and vice versa, and how to compare the two approaches.

Recent work on Winograd-based convolution allows for a great reduction of computational complexity, but existing implementations are limited to 2D data and a single kernel size of 3 by 3. They can achieve only slightly better, and often worse performance than better optimized, direct convolution implementations. We propose and implement an algorithm for N-dimensional Winograd-based convolution that allows arbitrary kernel sizes and is optimized for manycore CPUs. Our algorithm achieves high hardware utilization through a series of optimizations. Our experiments show that on modern ConvNets, our optimized implementation, is on average more than 3x, and sometimes 8x faster than other state-of-the-art CPU implementations on an Intel Xeon Phi manycore processors. Moreover, our implementation on the Xeon Phi achieves competitive performance for 2D ConvNets and superior performance for 3D ConvNets, compared with the best GPU implementations.

Convolutional nets have been shown to achieve state-of-the-art accuracy in many biomedical image analysis tasks. To deploy convolutional nets in practical working systems, it is also important to solve the efficient inference problem. Namely, one should be able to apply an already-trained convolutional network to many large images using limited computational resources. 3D images are especially relevant because biological tissues are 3D, and data volumes are typically high for 3D. While it is common to use GPUs for convolutional net inference, there may be environments where CPUs are more abundant or accessible. In this paper we present PZnet, a CPU-only engine that can be used to perform inference for a variety of 3D convolutional net architectures. PZNet outperforms MKL-based CPU implementations of PyTorch and Tensorflow by more than 3.5x for the popular 3D U-net architecture. Moreover, based on current pricing of preemptible or spot instances, cloud CPU inference with PZnet is competitive in cost with cloud GPU inference, for U-net style architectures.