Speech Brain-Computer Interfaces rely on decoding algorithms that transform neural activity into speech. A current challenge is to achieve intelligible speech synthesis in real time from continuous ongoing brain activity, ideally without the need of language models that prevent free-speech production. As a first step toward this goal, we introduce here an encoder-decoder architecture, in which neural data is first encoded into a latent space using a multi-layer vision transformer (ViT), and then these latent variables are converted into acoustic coefficients using a bidirectional LSTM recurrent network. This network is compared to a more conventional architecture where the encoding is performed using a convolutional neural network. Moreover, we introduce a new data-driven data augmentation strategy based on Dynamic Time Warping (DTW) to increase a training dataset based on the intrinsic variability of its input neural features. On two ECoG datasets obtained in participants performing an overt speech task, we found that ViT-encoding outperforms CNN-encoding to predict produced speech offline and that DTW-based data augmentation also improves decoding performance.

We show that the task of synthesizing human motion conditioned on a set of key frames can be solved more accurately and effectively if a deep learning based interpolator operates in the delta mode using the spherical linear interpolator as a baseline. We empirically demonstrate the strength of our approach on publicly available datasets achieving state-of-the-art performance. We further generalize these results by showing that the $\Delta$-regime is viable with respect to the reference of the last known frame (also known as the zero-velocity model). This supports the more general conclusion that operating in the reference frame local to input frames is more accurate and robust than in the global (world) reference frame advocated in previous work. Our code is publicly available at https://github.com/boreshkinai/delta-interpolator.

ABSTRACT Broca’s area has long been described as a central region of cortical speech networks. Yet, its detailed role during speech production remains largely unknown and even sometimes debated. Recordings at the individual neuron level could help understand whether and how speech is encoded in this region but such data remain very scarce. Here we use direct intracortical recording in pars triangularis of human Broca’s area to show that the encoding of speech in this region relies not only on spike rates but also on the precise timing of action potentials within individual spike trains. First, we found that the overall spike rate of the whole population remained unchanged between periods of rest, overt and covert speech, but that individual firing rates of a few neurons fluctuated across these conditions. These fluctuations resulted in different overall population dynamics across conditions. Next, we also observed that the temporal arrangement of spikes within individual spike trains was not purely random but also signed which sentence was produced. By matching ensemble spike trains based on inter-spike intervals, it was possible to decode which sentence had been pronounced well above chance and with a comparable accuracy than when using spike counts. Moreover, the temporal code characterizing the overt production of individual sentences was found to be largely conserved when the same sentences were covertly imagined and enabled to decode cover sentences with an accuracy 75% higher than when considering spike counts. Altogether, these results suggest the existence of 2 modes of speech encoding in Broca’s area, one based on the modulation of individual firing rates and the other based on the precise temporal structure of individual spike trains, and that the latter type of encoding remains more largely conserved between overt and covert speech production.

Most 3D games leverage animations to breathe life into characters. Even at the early stages of game development, prototyping with working animations helps developers quickly understand and adapt the game design. Controllable characters require animations that are constrained in multiple ways in order to blend together and avoid artifacts such as feet sliding. Authoring, modifying and iterating on such animations is a costly process that often requires high level expertise or expensive motion capture. We present a novel AI-powered graph tool that allows users to leverage generative machine learning models, presented as nodes in the graph, and create animations from only a few high-level inputs. This greatly reduces the time and expertise required to author animations from scratch, while offering unseen levels of control with generative motion creation. The combination of the many relevant nodes that the graph supports and the flexibility of our state-of-the-art diffusion models allows for going beyond text-to-motion and in-betweening, common in the literature. We can generate motions that can be synchronized on multiple aspects, such as duration, speed, feet contact timings, while being constrained by signals such as text-prompts and trajectories. Our diffusion models also allow for stylizing existing motions, and generating looping animations. This makes it possible, for the first time to our knowledge, to generate complete working locomotion sets from high level inputs such as text and duration, directly usable with common character controllers. Our system being implemented in Unity, we showcase the power of this novel approach by constructing a prototype animated character with working locomotion and attack motions without preexisting animations, in a few minutes.

Introduction:Speech BCIs aim at reconstructing speech in real time from ongoing cortical activity. Ideal BCIs would need to reconstruct speech audio signal frame by frame on a millisecond-timescale. Such approaches require fast computation. In this respect, linear decoder are good candidates and have been widely used in motor BCIs. Yet, they have been very seldomly studied for speech reconstruction, and never for reconstruction of articulatory movements from intracranial activity. Here, we compared vanilla linear regression, ridge-regularized linear regressions, and partial least squares regressions for offline decoding of overt speech from cortical activity.Methods:Two decoding paradigms were investigated: (1) direct decoding of acoustic vocoder features of speech, and (2) indirect decoding of vocoder features through an intermediate articulatory representation chained with a real-time-compatible DNN-based articulatory-to-acoustic synthesizer. Participant's articulatory trajectories were estimated from an electromagnetic-articulography dataset using dynamic time warping. The accuracy of the decoders was evaluated by computing correlations between original and reconstructed features.Results:We found that similar performance was achieved by all linear methods well above chance levels, albeit without reaching intelligibility. Direct and indirect methods achieved comparable performance, with an advantage for direct decoding.Discussion:Future work will address the development of an improved neural speech decoder compatible with fast frame-by-frame speech reconstruction from ongoing activity at a millisecond timescale.

Inverse Kinematics (IK) systems are often rigid with respect to their input character, thus requiring user intervention to be adapted to new skeletons. In this paper we aim at creating a flexible, learned IK solver applicable to a wide variety of human morphologies. We extend a state-of-the-art machine learning IK solver to operate on the well known Skinned Multi-Person Linear model (SMPL). We call our model SMPL-IK, and show that when integrated into real-time 3D software, this extended system opens up opportunities for defining novel AI-assisted animation workflows. For example, when chained with existing pose estimation algorithms, SMPL-IK accelerates posing by allowing users to bootstrap 3D scenes from 2D images while allowing for further editing. Additionally, we propose a novel SMPL Shape Inversion mechanism (SMPL-SI) to map arbitrary humanoid characters to the SMPL space, allowing artists to leverage SMPL-IK on custom characters. In addition to qualitative demos showing proposed tools, we present quantitative SMPL-IK baselines on the H36M and AMASS datasets. Our code is publicly available https://github.com/boreshkinai/smpl-ik.

Our work focuses on the development of a learnable neural representation of human pose for advanced AI assisted animation tooling. Specifically, we tackle the problem of constructing a full static human pose based on sparse and variable user inputs (e.g. locations and/or orientations of a subset of body joints). To solve this problem, we propose a novel neural architecture that combines residual connections with prototype encoding of a partially specified pose to create a new complete pose from the learned latent space. We show that our architecture outperforms a baseline based on Transformer, both in terms of accuracy and computational efficiency. Additionally, we develop a user interface to integrate our neural model in Unity, a real-time 3D development platform. Furthermore, we introduce two new datasets representing the static human pose modeling problem, based on high-quality human motion capture data, which will be released publicly along with model code.

We present a tool that allows users to quickly author character poses. Authoring character poses is typically done by professional artists and is a time consuming process that involves a lot of user manipulations. Our tool leverages both machine learning and a physics engine to enable users with no artistic experience to author natural-looking poses in a few seconds. First, we trained a machine learning (ML) model to predict a full character pose, including individual fingers, from a set of sparse constraints. These constraints allow the user to control the final pose by specifying final joint positions, orientations or a target that they should face. Our ML architecture allows the constraints to be given in any order and number. The model was trained on a large set of motion capture data so that it predicts natural and realistic human poses. Second, we integrated our ML model with a physics solver so that the final pose also respects environmental constraints such as colliding with other objects. This allows the user to quickly pose a character interacting with the environment, another character or itself. Finally, we developed a user-friendly interface to control this tool. We believe that the combination of machine learning and physics lower the entry bar to character animation.

Our work focuses on the development of a learnable neural representation of human pose for advanced AI assisted animation tooling. Specifically, we tackle the problem of constructing a full static human pose based on sparse and variable user inputs (e.g. locations and/or orientations of a subset of body joints). To solve this problem, we propose a novel neural architecture that combines residual connections with prototype encoding of a partially specified pose to create a new complete pose from the learned latent space. We show that our architecture outperforms a baseline based on Transformer, both in terms of accuracy and computational efficiency. Additionally, we develop a user interface to integrate our neural model in Unity, a real-time 3D development platform. Furthermore, we introduce two new datasets representing the static human pose modeling problem, based on high-quality human motion capture data, which will be released publicly along with model code.

Objective. A current challenge of neurotechnologies is to develop speech brain-computer interfaces aiming at restoring communication in people unable to speak. To achieve a proof of concept of such system, neural activity of patients implanted for clinical reasons can be recorded while they speak. Using such simultaneously recorded audio and neural data, decoders can be built to predict speech features using features extracted from brain signals. A typical neural feature is the spectral power of field potentials in the high-gamma frequency band, which happens to overlap the frequency range of speech acoustic signals, especially the fundamental frequency of the voice. Here, we analyzed human electrocorticographic and intracortical recordings during speech production and perception as well as a rat microelectrocorticographic recording during sound perception. We observed that several datasets, recorded with different recording setups, contained spectrotemporal features highly correlated with those of the sound produced by or delivered to the participants, especially within the high-gamma band and above, strongly suggesting a contamination of electrophysiological recordings by the sound signal. This study investigated the presence of acoustic contamination and its possible source. Approach. We developed analysis methods and a statistical criterion to objectively assess the presence or absence of contamination-specific correlations, which we used to screen several datasets from five centers worldwide. Main results. Not all but several datasets, recorded in a variety of conditions, showed significant evidence of acoustic contamination. Three out of five centers were concerned by the phenomenon. In a recording showing high contamination, the use of high-gamma band features dramatically facilitated the performance of linear decoding of acoustic speech features, while such improvement was very limited for another recording showing no significant contamination. Further analysis and in vitro replication suggest that the contamination is caused by the mechanical action of the sound waves onto the cables and connectors along the recording chain, transforming sound vibrations into an undesired electrical noise affecting the biopotential measurements. Significance. Although this study does not per se question the presence of speech-relevant physiological information in the high-gamma range and above (multiunit activity), it alerts on the fact that acoustic contamination of neural signals should be proofed and eliminated before investigating the cortical dynamics of these processes. To this end, we make available a toolbox implementing the proposed statistical approach to quickly assess the extent of contamination in an electrophysiological recording (https://doi.org/10.5281/zenodo.3929296).

A current challenge of neurotechnologies is the development of speech brain-computer interfaces to restore communication in people unable to speak. To achieve a proof of concept of such system, neural activity of patients implanted for clinical reasons can be recorded while they speak. Using such simultaneously recorded audio and neural data, decoders can be built to predict speech features using features extracted from brain signals. A typical neural feature is the spectral power of field potentials in the high-gamma frequency band (between 70 and 200 Hz), a range that happens to overlap the fundamental frequency of speech. Here, we analyzed human electrocorticographic (ECoG) and intracortical recordings during speech production and perception as well as rat microelectrocorticographic (µ-ECoG) recordings during sound perception. We observed that electrophysiological signals, recorded with different recording setups, often contain spectrotemporal features highly correlated with those of the sound, especially within the high-gamma band. The characteristics of these correlated spectrotemporal features support a contamination of electrophysiological recordings by sound. In a recording showing high contamination, using neural features within the high-gamma frequency band dramatically increased the performance of linear decoding of acoustic speech features, while such improvement was very limited for another recording showing weak contamination. Further analysis and in vitro replication suggest that the contamination is caused by a mechanical action of the sound waves onto the cables and connectors along the recording chain, transforming sound vibrations into an undesired electrical noise that contaminates the biopotential measurements. This study does not question the existence of relevant physiological neural information underlying speech production or sound perception in the high-gamma frequency band, but alerts on the fact that care should be taken to evaluate and eliminate any possible acoustic contamination of neural signals in order to investigate the cortical dynamics of these processes.

Restoring communication in case of aphasia is a key challenge for neurotechnologies. To this end, brain-computer strategies can be envisioned to allow artificial speech synthesis from the continuous decoding of neural signals underlying speech imagination. Such speech brain-computer interfaces do not exist yet and their design should consider three key choices that need to be made: the choice of appropriate brain regions to record neural activity from, the choice of an appropriate recording technique, and the choice of a neural decoding scheme in association with an appropriate speech synthesis method. These key considerations are discussed here in light of (1) the current understanding of the functional neuroanatomy of cortical areas underlying overt and covert speech production, (2) the available literature making use of a variety of brain recording techniques to better characterize and address the challenge of decoding cortical speech signals, and (3) the different speech synthesis approaches that can be considered depending on the level of speech representation (phonetic, acoustic or articulatory) envisioned to be decoded at the core of a speech BCI paradigm.

En France, 300 000 personnes souffrent d’un trouble de la parole et plus de 5 millions de par le monde. Ces troubles surviennent le plus souvent suite à un accident vasculaire cérébral ayant endommagé le réseau cortical du langage. Cependant, des aphasies sévères sont également observées chez des personnes où ce réseau n’a pas été endommagé, notamment dans le cas d’un syndrome d’enfermement (« locked-in »), de maladies neurodégénératives telles que la sclérose latérale amyotrophique ou la maladie de Parkinson, de myopathies, ou encore de comas. Dans ce cas, il est possible d’envisager de restaurer la parole par un paradigme d’interface cerveau–machine (BCI) où les activités corticales sont décodées pour prédire ce que souhaite dire la personne [3]. Dans cette perspective, nous développons une stratégie visant à enregistrer les aires motrices de la parole contrôlant les articulateurs du conduit vocal, et à décoder leurs activités pour contrôler un synthétiseur vocal articulatoire. Les différentes étapes de ce projet seront discutées, ainsi que les premiers résultats déjà obtenus en termes de synthèse vocale artificielle en temps réel [1] et de décodage des activités corticales des aires de la parole. Nous soulignerons également certaines questions éthiques associées au projet et plus largement au développement de neuroprothèses et d’interfaces cerveau-machine en général [2].

Restoring natural speech in paralyzed and aphasic people could be achieved using a Brain-Computer Interface (BCI) controlling a speech synthesizer in real-time. To reach this goal, a prerequisite is to develop a speech synthesizer producing intelligible speech in real-time with a reasonable number of control parameters. We present here an articulatory-based speech synthesizer that can be controlled in real-time for future BCI applications. This synthesizer converts movements of the main speech articulators (tongue, jaw, velum, and lips) into intelligible speech. The articulatory-to-acoustic mapping is performed using a deep neural network (DNN) trained on electromagnetic articulography (EMA) data recorded on a reference speaker synchronously with the produced speech signal. This DNN is then used in both offline and online modes to map the position of sensors glued on different speech articulators into acoustic parameters that are further converted into an audio signal using a vocoder. In offline mode, highly intelligible speech could be obtained as assessed by perceptual evaluation performed by 12 listeners. Then, to anticipate future BCI applications, we further assessed the real-time control of the synthesizer by both the reference speaker and new speakers, in a closed-loop paradigm using EMA data recorded in real time. A short calibration period was used to compensate for differences in sensor positions and articulatory differences between new speakers and the reference speaker. We found that real-time synthesis of vowels and consonants was possible with good intelligibility. In conclusion, these results open to future speech BCI applications using such articulatory-based speech synthesizer.

This article presents a pilot study on the real-time control of an articulatory synthesizer based on deep neural network (DNN), in the context of silent speech interface. The underlying hypothesis is that a silent speaker could benefit from real-time audio feedback to regulate his/her own production. In this study, we use 3D electromagnetic-articulography (EMA) to capture speech articulation, a DNN to convert EMA to spectral trajectories in real-time, and a standard vocoder excited by white noise for audio synthesis. As shown by recent literature on silent speech, adaptation of the articulo-acoustic modeling process is needed to account for possible inconsistencies between the initial training phase and practical usage conditions. In this study, we focus on different sensor setups across sessions (for the same speaker). Model adaptation is performed by cascading another neural network to the DNN used for articulatory-to-acoustic mapping. The intelligibility of the synthetic speech signal converted in real-time is evaluated using both objective and perceptual measurements.

This paper describes a machine learning approach for extracting automatically the tongue contour in ultrasound images. This method is developed in the context of visual articulatory biofeedback for speech therapy. The goal is to provide a speaker with an intuitive visualization of his/her tongue movement, in real-time, and with minimum human intervention. Contrary to most widely used techniques based on active contours, the proposed method aims at exploiting the information of all image pixels to infer the tongue contour. For that purpose, a compact representation of each image is extracted using a PCA-based decomposition technique (named EigenTongue). Artificial neural networks are then used to convert the extracted visual features into control parameters of a PCA-based tongue contour model. The proposed method is evaluated on 9 speakers, using data recorded with the ultrasound probe hold manually (as in the targeted application). Speaker-dependent experiments demonstrated the effectiveness of the proposed method (with an average error of ~1.3 mm when training from 80 manually annotated images), even when the tongue contour is poorly imaged. The performance was significantly lower in speaker-independent experiments (i.e. when estimating contours on an unknown speaker), likely due to anatomical differences across speakers.

Brain-Computer Interfaces (BCIs) usually propose typing strategies to restore communication for paralyzed and aphasic people. A more natural way would be to use speech BCI directly controlling a speech synthesizer. Toward this goal, a prerequisite is the development a synthesizer that should i) produce intelligible speech, ii) run in real time, iii) depend on as few parameters as possible, and iv) be robust to error fluctuations on the control parameters. In this context, we describe here an articulatory-to-acoustic mapping approach based on deep neural network (DNN) trained on electromagnetic articulography (EMA) data recorded synchronously with produced speech sounds. On this corpus, the DNN-based model provided a speech synthesis quality (as assessed by automatic speech recognition and behavioral testing) comparable to a state-of-the-art Gaussian mixture model (GMM), yet showing higher robustness when noise was added to the EMA coordinates. Moreover, to envision BCI applications, this robustness was also assessed when the space covered by the 12 original articulatory parameters was reduced to 7 parameters using deep auto-encoders (DAE). Given that this method can be implemented in real time, DNN-based articulatory speech synthesis seems a good candidate for speech BCI applications. Index Terms: articulatory speech synthesis, brain computer interface (BCI), deep neural networks, deep auto-encoder, EMA, noise robustness, dimensionality reduction

Vers une interface cerveau-machine pour la restauration de la parole Restorer la faculté de parler chez des personnes paralysées et aphasiques pourrait être envisagée via l’utilisation d’une interface cerveau-machine permettant de contrôler un synthétiseur de parole en temps réel. L’objectif de cette thèse était de développer trois aspects nécessaires à la mise au point d’une telle preuve de concept.Premièrement, un synthétiseur permettant de produire en temps-réel de la parole intelligible et controlé par un nombre raisonable de paramètres est nécessaire. Nous avons choisi de synthétiser de la parole à partir des mouvements des articulateurs du conduit vocal. En effet, des études récentes ont suggéré que l’activité neuronale du cortex moteur de la parole pourrait contenir suffisamment d’information pour décoder la parole, et particulièrement ses propriété articulatoire (ex. l’ouverture des lèvres). Nous avons donc développé un synthétiseur produisant de la parole intelligible à partir de données articulatoires. Dans un premier temps, nous avons enregistré un large corpus de données articulatoire et acoustiques synchrones chez un locuteur. Ensuite, nous avons utilisé des techniques d’apprentissage automatique, en particulier des réseaux de neurones profonds, pour construire un modèle permettant de convertir des données articulatoires en parole. Ce synthétisuer a été construit pour fonctionner en temps réel. Enfin, comme première étape vers un contrôle neuronal de ce synthétiseur, nous avons testé qu’il pouvait être contrôlé en temps réel par plusieurs locuteurs, pour produire de la parole inetlligible à partir de leurs mouvements articulatoires dans un paradigme de boucle fermée.Deuxièmement, nous avons étudié le décodage de la parole et de ses propriétés articulatoires à partir d’activités neuronales essentiellement enregistrées dans le cortex moteur de la parole. Nous avons construit un outil permettant de localiser les aires corticales actives, en ligne pendant des chirurgies éveillées à l’hôpital de Grenoble, et nous avons testé ce système chez deux patients atteints d’un cancer du cerveau. Les résultats ont montré que le cortex moteur exhibe une activité spécifique pendant la production de parole dans les bandes beta et gamma du signal, y compris lors de l’imagination de la parole. Les données enregistrées ont ensuite pu être analysées pour décoder l’intention de parler du sujet (réelle ou imaginée), ainsi que la vibration des cordes vocales et les trajectoires des articulateurs principaux du conduit vocal significativement au dessus du niveau de la chance.Enfin, nous nous sommes intéressés aux questions éthiques qui accompagnent le développement et l’usage des interfaces cerveau-machine. Nous avons en particulier considéré trois niveaux de réflexion éthique concernant respectivement l’animal, l’humain et l’humanité.