Next-generation brain-computer interfaces will enable motor and speech decoding in humans [1]–[3] and improve our understanding of brain function [4]. To achieve this requires high-density multi-electrode arrays (HD-MEA) [5], [6]. This leads to massive amounts of raw data that must be reduced on-chip to enable wireless operation [7]. Spike sorting (SS) assigns spikes to putative neurons and can reduce the data rate substantially because only the neuron ID needs to be transmitted when a spike occurs. Prior art focuses on improving the scalability and power efficiency of on-chip SS [8]–[15]. However, they either require a large input buffer [12]–[14], use temporal features (TF) that do not scale well to multi-channel systems [8]–[13], access the entire clustering memory for every spike [11]–[13], or use high power [8]–[11] and area [8]–[10]. This work uses event-driven spike detection and spatial spike sorting to deal with these challenges and achieve 74nW/ch, 0.00029mm2/ch, and <50μs latency with 1024 channels. This represents a >10x improvement in power and area efficiency with 3x more channels than prior art (Fig. 15.2.6).

Objective.Identifying neuronal cell types and their biophysical properties based on their extracellular electrical features is a major challenge for experimental neuroscience and for the development of high-resolution brain-machine interfaces. One example is identification of retinal ganglion cell (RGC) types and their visual response properties, which is fundamental for developing future electronic implants that can restore vision.Approach.The electrical image (EI) of a RGC, or the mean spatio-temporal voltage footprint of its recorded spikes on a high-density electrode array, contains substantial information about its anatomical, morphological, and functional properties. However, the analysis of these properties is complex because of the high-dimensional nature of the EI. We present a novel optimization-based algorithm to decompose EI into a low-dimensional, biophysically-based representation: the temporally-shifted superposition of three learned basis waveforms corresponding to spike waveforms produced in the somatic, dendritic and axonal cellular compartments.Main results.The decomposition was evaluated using large-scale multi-electrode recordings from the macaque retina. The decomposition accurately localized the somatic and dendritic compartments of the cell. The imputed dendritic fields of RGCs correctly predicted the location and shape of their visual receptive fields. The inferred waveform amplitudes and shapes accurately identified the four major primate RGC types (ON and OFF midget and parasol cells) substantially more accurately than previous approaches.Significance.These findings contribute to more accurate inference of RGC types and their original light responses based purely on their electrical features, with potential implications for vision restoration technology.

Objective.Neural interfaces are designed to evoke specific patterns of electrical activity in populations of neurons by stimulating with many electrodes. However, currents passed simultaneously through multiple electrodes often combine nonlinearly to drive neural responses, making evoked responses difficult to predict and control. This response nonlinearity could arise from the interaction of many excitable sites in each cell, any of which can produce a spike. However, this multi-site activation hypothesis is difficult to verify experimentally.Approach.We developed a biophysical model to study retinal ganglion cell responses to multi-electrode stimulation and validated it using data collected fromex vivopreparations of the macaque retina using a microelectrode array (512 electrodes; 30µm pitch; 10µm diameter).Results.First, the model was validated by using it to reproduce essential empirical findings from single-electrode recording and stimulation, including recorded spike voltage waveforms at multiple locations and sigmoidal responses to injected current. Then, stimulation with two electrodes was modeled to test how the positioning of the electrodes relative to the cell affected the degree of response nonlinearity. Currents passed through pairs of electrodes positioned near the cell body or far from the axon (>40µm) exhibited approximately linear summation in evoking spikes. Currents passed through pairs of electrodes close to the axon summed linearly when their locations along the axon were similar, and nonlinearly otherwise. Over a range of electrode placements, several distinct, localized spike initiation sites were observed, and the number of these sites covaried with the degree of response nonlinearity. Similar trends were observed for three-electrode stimuli. All of these trends in the simulation were consistent with experimental observations.Significance.These findings support the multi-site activation hypothesis for nonlinear activation of neurons, providing a biophysical interpretation of previous experimental results and potentially enabling more efficient use of multi-electrode stimuli in future neural implants.

Silicon-based planar microelectronics is a powerful tool for scalably recording and modulating neural activity at high spatiotemporal resolution, but it remains challenging to target neural structures in three dimensions (3D). We present a method for directly fabricating 3D arrays of tissue-penetrating microelectrodes onto silicon microelectronics. Leveraging a high-resolution 3D printing technology based on 2-photon polymerization and scalable microfabrication processes, we fabricated arrays of 6,600 microelectrodes 10-130 µm tall and at 35-μm pitch onto a planar silicon-based microelectrode array. The process enables customizable electrode shape, height and positioning for precise targeting of neuron populations distributed in 3D. As a proof of concept, we addressed the challenge of specifically targeting retinal ganglion cell (RGC) somas when interfacing with the retina. The array was customized for insertion into the retina and recording from somas while avoiding the axon layer. We verified locations of the microelectrodes with confocal microscopy and recorded high-resolution spontaneous RGC activity at cellular resolution. This revealed strong somatic and dendritic components with little axon contribution, unlike recordings with planar microelectrode arrays. The technology could be a versatile solution for interfacing silicon microelectronics with neural structures and modulating neural activity at large scale with single-cell resolution.

Neural implants have the potential to restore lost sensory function by electrically evoking the complex naturalistic activity patterns of neural populations. However, it can be difficult to predict and control evoked neural responses to simultaneous multi-electrode stimulation due to nonlinearity of the responses. We present a solution to this problem and demonstrate its utility in the context of a bidirectional retinal implant for restoring vision. A dynamically optimized stimulation approach encodes incoming visual stimuli into a rapid, greedily chosen, temporally dithered and spatially multiplexed sequence of simple stimulation patterns. Stimuli are selected to optimize the reconstruction of the visual stimulus from the evoked responses. Temporal dithering exploits the slow time scales of downstream neural processing, and spatial multiplexing exploits the independence of responses generated by distant electrodes. The approach was evaluated using an experimental laboratory prototype of a retinal implant: large-scale, high-resolution multi-electrode stimulation and recording of macaque and rat retinal ganglion cells ex vivo. The dynamically optimized stimulation approach substantially enhanced performance compared to existing approaches based on static mapping between visual stimulus intensity and current amplitude. The modular framework enabled parallel extensions to naturalistic viewing conditions, incorporation of perceptual similarity measures, and efficient implementation for an implantable device. A direct closed-loop test of the approach supported its potential use in vision restoration.

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulusevoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Objective. Bi-directional electronic neural interfaces, capable of both electrical recording and stimulation, communicate with the nervous system to permit precise calibration of electrical inputs by capturing the evoked neural responses. However, one significant challenge is that stimulation artifacts often mask the actual neural signals. To address this issue, we introduce a novel approach that employs dynamical control systems to detect and decipher electrically evoked neural activity despite the presence of electrical artifacts.Approach. Our proposed method leverages the unique spatiotemporal patterns of neural activity and electrical artifacts to distinguish and identify individual neural spikes. We designed distinctive dynamical models for both the stimulation artifact and each neuron observed during spontaneous neural activity. We can estimate which neurons were active by analyzing the recorded voltage responses across multiple electrodes post-stimulation. This technique also allows us to exclude signals from electrodes heavily affected by stimulation artifacts, such as the stimulating electrode itself, yet still accurately differentiate between evoked spikes and electrical artifacts.Main results. We applied our method to high-density multi-electrode recordings from the primate retina in anex vivosetup, using a grid of 512 electrodes. Through repeated electrical stimulations at varying amplitudes, we were able to construct activation curves for each neuron. The curves obtained with our method closely resembled those derived from manual spike sorting. Additionally, the stimulation thresholds we estimated strongly agreed with those determined through manual analysis, demonstrating high reliability (R2=0.951for human 1 andR2=0.944for human 2).Significance. Our method can effectively separate evoked neural spikes from stimulation artifacts by exploiting the distinct spatiotemporal propagation patterns captured by a dense, large-scale multi-electrode array. This technique holds promise for future applications in real-time closed-loop stimulation systems and for managing multi-channel stimulation strategies.

This paper presents a neural recording IC featuring lossy compression during digitization, thus preventing data deluge and enabling a compact active digital pixel design. The wired-OR-based compression discards unwanted baseline samples while allowing the reconstruction of spike samples. The IC features a 32x32 MEA with $36 \mu m$ pixel pitch and consumes 268nW per pixel from a single 1V supply. It achieves $9.8 \mu V_{RMS}$ input-referred noise and 0.3-5kHz bandwidth, resulting in NEF/PEF of 3.7/14.1.

Linear-nonlinear (LN) cascade models provide a simple way to capture retinal ganglion cell (RGC) responses to artificial stimuli such as white noise, but their ability to model responses to natural images is limited. Recently, convolutional neural network (CNN) models have been shown to produce light response predictions that were substantially more accurate than those of a LN model. However, this modeling approach has not yet been applied to responses of macaque or human RGCs to natural images. Here, we train and test a CNN model on responses to natural images of the four numerically dominant RGC types in the macaque and human retina - ON parasol, OFF parasol, ON midget and OFF midget cells. Compared with the LN model, the CNN model provided substantially more accurate response predictions. Linear reconstructions of the visual stimulus were more accurate for CNN compared to LN model-generated responses, relative to reconstructions obtained from the recorded data. These findings demonstrate the effectiveness of a CNN model in capturing light responses of major RGC types in the macaque and human retinas in natural conditions.

Retinal implants are designed to restore vision to people blinded by photoreceptor degeneration by evoking spiking activity in surviving retinal ganglion cells (RGCs) and thus conveying artificial visual signals to the brain. One approach is to implant a high-density micro-electrode array on the surface of the retina and apply electrical stimuli to reproduce the rich spatiotemporal patterns of visually evoked RGC responses. However, this requires stimulation through multiple electrodes, which is complicated by the non-linear superposition of single-electrode responses. A recently published approach to this problem, greedy temporal dithering (GTD), optimally approximates the visually evoked activity by rapidly interleaving stimulation at different locations on the array. However, the GTD algorithm is too computationally demanding to use with an implantable chip in real-time with present-day hardware, and it scales poorly with the number of cells addressed. Here we present a computationally efficient implementation of the GTD algorithm by exploiting the fundamental anatomical and morphological properties of the underlying neural circuitry. Efficiency is achieved by partitioning distinct groups of cells according to their overlapping sensitivity to electrical stimulation, which in turn allows for efficient parallel stimulation. Using high-density multi-electrode recording and stimulation in the primate retina as a lab prototype for a future implant, we demonstrate a 10-fold decrease in GTD active compute time using the partitioning scheme with less than a 10% reduction in the accuracy of visual reconstruction.

Epiretinal implants are designed to restore visual function by direct stimulation of retinal ganglion cells (RGCs) with a multi-electrode array. However, the efficacy of present-day epiretinal implants is limited by indiscriminate, simultaneous activation of many RGCs of different types that normally convey distinct visual information, resulting in highly unnatural visual signals. Even with high-density multi-electrode arrays in a laboratory setting, single-electrode stimulation often cannot target an individual cell selectively. A possible solution is to use spatially patterned multi-electrode stimulation. Previous studies have demonstrated enhanced selectivity using such stimuli, but a general framework for characterizing the responses of RGCs to stimulation by multiple electrodes has yet to be realized. Here we propose a generalizable model for the response of a neuron to multi-electrode stimuli, and demonstrate that it can be used to systematically and efficiently improve the selectivity of epiretinal stimulation. The efficacy of the model and of multi-electrode stimuli is validated using three-electrode triplet stimulation. The model assumes that there are several possible activation sites on each cell where a spike can originate, and that the probability of activation at each site is determined by a weighted sum of the currents applied on the electrodes. The multi-site model provides excellent fits to experimental data as well as an interpretable mechanism of multi-electrode activation. Additionally, since the model has few parameters, it permits efficient closed-loop calibration. An optimal experimental design is presented and validated for quickly learning the model parameters and identifying multi-electrode stimuli that are more selective than can be achieved with single-electrode stimuli. The model and closed-loop algorithm could potentially be used to optimize the selectivity of other neural interfaces.

Future high-density and high channel count neural interfaces that enable simultaneous recording of tens of thousands of neurons will provide a gateway to study, restore and augment neural functions. However, building such technology within the bit-rate limit and power budget of a fully implantable device is challenging. The wired-OR compressive readout architecture addresses the data deluge challenge of a high channel count neural interface using lossy compression at the analog-to-digital interface. In this article, we assess the suitability of wired-OR for several steps that are important for neuroengineering, including spike detection, spike assignment and waveform estimation. For various wiring configurations of wired-OR and assumptions about the quality of the underlying signal, we characterize the trade-off between compression ratio and task-specific signal fidelity metrics. Using data from 18 large-scale microelectrode array recordings in macaque retina ex vivo, we find that for an event SNR of 7-10, wired-OR correctly detects and assigns at least 80% of the spikes with at least 50× compression. The wired-OR approach also robustly encodes action potential waveform information, enabling downstream processing such as cell-type classification. Finally, we show that by applying an LZ77-based lossless compressor (gzip) to the output of the wired-OR architecture, 1000× compression can be achieved over the baseline recordings.

Electrical stimulation of retinal ganglion cells (RGCs) with electronic implants provides rudimentary artificial vision to people blinded by retinal degeneration. However, current devices stimulate indiscriminately and therefore cannot reproduce the intricate neural code of the retina. Recent work has demonstrated more precise activation of RGCs using focal electrical stimulation with multielectrode arrays in the peripheral macaque retina, but it is unclear how effective this can be in the central retina, which is required for high-resolution vision. This work probes the neural code and effectiveness of focal epiretinal stimulation in the central macaque retina, using large-scale electrical recording and stimulationex vivo. The functional organization, light response properties, and electrical properties of the major RGC types in the central retina were mostly similar to the peripheral retina, with some notable differences in density, kinetics, linearity, spiking statistics, and correlations. The major RGC types could be distinguished by their intrinsic electrical properties. Electrical stimulation targeting parasol cells revealed similar activation thresholds and reduced axon bundle activation in the central retina, but lower stimulation selectivity. Quantitative evaluation of the potential for image reconstruction from electrically evoked parasol cell signals revealed higher overall expected image quality in the central retina. An exploration of inadvertent midget cell activation suggested that it could contribute high spatial frequency noise to the visual signal carried by parasol cells. These results support the possibility of reproducing high-acuity visual signals in the central retina with an epiretinal implant.SIGNIFICANCE STATEMENTArtificial restoration of vision with retinal implants is a major treatment for blindness. However, present-day implants do not provide high-resolution visual perception, in part because they do not reproduce the natural neural code of the retina. Here, we demonstrate the level of visual signal reproduction that is possible with a future implant by examining how accurately responses to electrical stimulation of parasol retinal ganglion cells can convey visual signals. Although the precision of electrical stimulation in the central retina was diminished relative to the peripheral retina, the quality of expected visual signal reconstruction in parasol cells was greater. These findings suggest that visual signals could be restored with high fidelity in the central retina using a future retinal implant.

High-fidelity electronic implants can in principle restore the function of neural circuits by precisely activating neurons via extracellular stimulation. However, direct characterization of the individual electrical sensitivity of a large population of target neurons, to precisely control their activity, can be difficult or impossible. A potential solution is to leverage biophysical principles to infer sensitivity to electrical stimulation from features of spontaneous electrical activity, which can be recorded relatively easily. Here, this approach is developed and its potential value for vision restoration is tested quantitatively using large-scale multielectrode stimulation and recording from retinal ganglion cells (RGCs) of male and female macaque monkeys ex vivo . Electrodes recording larger spikes from a given cell exhibited lower stimulation thresholds across cell types, retinas, and eccentricities, with systematic and distinct trends for somas and axons. Thresholds for somatic stimulation increased with distance from the axon initial segment. The dependence of spike probability on injected current was inversely related to threshold, and was substantially steeper for axonal than somatic compartments, which could be identified by their recorded electrical signatures. Dendritic stimulation was largely ineffective for eliciting spikes. These trends were quantitatively reproduced with biophysical simulations. Results from human RGCs were broadly similar. The inference of stimulation sensitivity from recorded electrical features was tested in a data-driven simulation of visual reconstruction, revealing that the approach could significantly improve the function of future high-fidelity retinal implants. SIGNIFICANCE STATEMENT This study demonstrates that individual in situ primate retinal ganglion cells of different types respond to artificially generated, external electrical fields in a systematic manner, in accordance with theoretical predictions, that allows for prediction of electrical stimulus sensitivity from recorded spontaneous activity. It also provides evidence that such an approach could be immensely helpful in the calibration of clinical retinal implants.

Objective. Retinal implants are designed to stimulate retinal ganglion cells (RGCs) in a way that restores sight to individuals blinded by photoreceptor degeneration. Reproducing high-acuity vision with these devices will likely require inferring the natural light responses of diverse RGC types in the implanted retina, without being able to measure them directly. Here we demonstrate an inference approach that exploits intrinsic electrophysiological features of primate RGCs.Approach.First, ON-parasol and OFF-parasol RGC types were identified using their intrinsic electrical features in large-scale multi-electrode recordings from macaque retina. Then, the electrically inferred somatic location, inferred cell type, and average linear-nonlinear-Poisson model parameters of each cell type were used to infer a light response model for each cell. The accuracy of the cell type classification and of reproducing measured light responses with the model were evaluated.Main results.A cell-type classifier trained on 246 large-scale multi-electrode recordings from 148 retinas achieved 95% mean accuracy on 29 test retinas. In five retinas tested, the inferred models achieved an average correlation with measured firing rates of 0.49 for white noise visual stimuli and 0.50 for natural scenes stimuli, compared to 0.65 and 0.58 respectively for models fitted to recorded light responses (an upper bound). Linear decoding of natural images from predicted RGC activity in one retina showed a mean correlation of 0.55 between decoded and true images, compared to an upper bound of 0.81 using models fitted to light response data.Significance.These results suggest that inference of RGC light response properties from intrinsic features of their electrical activity may be a useful approach for high-fidelity sight restoration. The overall strategy of first inferring cell type from electrical features and then exploiting cell type to help infer natural cell function may also prove broadly useful to neural interfaces.