NEMO-BMI: Auto-adaptive NEuroMOrphic
Brain-Machine Interface: toward
fully-embedded neuroprosthetics
The NEMO-BMI project aims at developing disruptive miniaturized Brain Machine Interfaces to restore mobility for patients with chronic spinal cord injury in their daily life.
Funded by European Innovative Council (pathfinder challenge project).
PROJECT OVERVIEW
Nearly 746,000 people sustain a spinal cord injury every year, with dramatic human, societal, and economic costs, leading to impairment or complete loss of motor functions. Motor Brain-Machine Interfaces (BMIs) translate brain neural signals into commands to external effectors. BMIs raise hopes that limb mobility may be restored, providing patients with control over orthoses, prostheses, or over their own limbs using electrical stimulation. The NEMO BMI project will conduct the exploration of assistance-free and easy-to-use portable neuroprosthetics by leveraging two implantable breakthrough technologies.This includes a wireless neuronal activity recorder, a real-time neuronal activity decoder based on integrated technologies, and a spinal cord stimulator.
To address the need of the largest number of patients with SCI, the next generation of BMI needs to be compatible with daily, autonomous use.
Motor Brain Machine Interfaces (BMIs) aim at translating brain neural signals into commands to robotic [1] or implanted electrical stimulator [2][3] effectors. The ongoing clinical trials carried out by EPFL, and CEA (STIMO-BSI – NCT04632290, and ‘BCI & Tetraplegia’ – NCT02550522) raise great hopes for SCI patients. They effectively assess the feasibility of chronic motor BMIs, based on the WIMAGINE® wireless Electrocorticogram (ECoG) recording implant, for long-term use in daily life.
Epidural spinal stimulation aims to translate the commands into electrical stimulator effectors. ONWARD has developed ARCIM Therapy, an implantable medical grade neurostimulation platform with unique real-time control capabilities. This platform includes a implantable pulse generator (IPG) that enables real-time control of 16 stimulation channels.
In the NEMO-BMI project, we will develop new auto-adaptive algorithms for brain decoding and spinal cord stimulation that will significantly contribute to enhancing knowledge on brain adaptation mechanisms. We foresee novelty in the design and implementation of neuromorphic hardware to sustain fast, secure, miniaturized and low-power neuroprosthetics.
Consortium Members and Expertise
Commissariat à l’Énergie Atomique et aux Énergies alternatives (CEA)
Clinatec team from CEA-Leti, will bring its expertise in implantable wireless ECoG recorder and Machine Learning decoding algorithms.
CEA-List will contribute to the project by fastening the decoding algorithms and by designing and manufacturing a silicon chip for neural signals decoding [4][5].
Illustration of the BCI technology developed by CEA
Ecole Polytechnique Fédérale de Lausanne (EPFL)
EPFL will bring its outstanding expertise on spinal cord stimulation to optimize muscle response and motion to restore motor functions [2].
Illustration of the EES technology developed by EPFL. Each muscle of the body can be accessed through the activation of a region of the spinal cord through electrical stimulation [6]
ONWARD medical
ONWARD will bring its’ advanced implantable spinal cord stimulators to restore mobility in individuals with chronic SCI.
ONWARD is a medical technology company creating innovative therapies to restore movement, independence, and health in people with spinal cord injuries. ONWARD’s work builds on more than a decade of basic science and preclinical research conducted at the world’s leading neuroscience laboratories. ONWARD’s ARC Therapy, which can be delivered by implantable (ARCIM) or external (ARCEX) systems, is designed to deliver targeted, programmed stimulation of the spinal cord to restore movement and other functions in people with spinal cord injury, ultimately improving their quality of life. ONWARD has received seven Breakthrough Device Designations from the FDA encompassing both ARCIM and ARCEX. The company’s first FDA pivotal trial, called Up-LIFT, completed enrollment in December 2021 with 65 subjects worldwide.
ONWARD is headquartered in Eindhoven, the Netherlands. It maintains a Science and Engineering Center in Lausanne, Switzerland, and has a growing US presence in Boston, Massachusetts. For additional information about the company, please visit ONWD.com
Illustration techno ONWARD: ARCIM Implantable neuromodulation device
Institute Of Information And Communication Technologies from the Bulgarian Academy of Sciences (IICT-BAS)
IICT will bring its expertise in neuromorphic [7] and reservoir [8] computing to investigate new paradigms of decoding algorithm.
Illustration of IICT neuromorphic reservoir decoder.
References
[1] A. L. Benabid et al., “An exoskeleton controlled by an epidural wireless brain–machine interface in a tetraplegic patient: a proof-of-concept demonstration,” Lancet Neurol., Oct. 2019, doi: 10.1016/S1474-4422(19)30321-7.
[2] F. B. Wagner et al., “Targeted neurotechnology restores walking in humans with spinal cord injury,” Nature, vol. 563, no. 7729, Art. no. 7729, Nov. 2018, doi: 10.1038/s41586-018-0649-2.
[3] Lorach, H., Charvet, G., Bloch, J., & Courtine, G. (2022). Brain-spine interfaces to reverse paralysis. National Science Review.
[4] Christensen, D. V., Dittmann, R., Linares-Barranco, B., Sebastian, A., Le Gallo, M., Redaelli, A., … & Pryds, N. (2022). 2022 roadmap on neuromorphic computing and engineering. Neuromorphic Computing and Engineering, 2(2), 022501.
[5] Miro-Panades, I., Tain, B., Christmann, J. F., Coriat, D., Lemaire, R., Jany, C., … & Clermidy, F. (2022). SamurAI: A Versatile IoT Node With Event-Driven Wake-Up and Embedded ML Acceleration. IEEE Journal of Solid-State Circuits.
[6] Rowald, A., Komi, S., Demesmaeker, R., Baaklini, E., Hernandez-Charpak, S. D., Paoles, E., … & Courtine, G. (2022). Activity-dependent spinal cord neuromodulation rapidly restores trunk and leg motor functions after complete paralysis. Nature medicine, 28(2), 260-271.
[7] N. K. Kasabov, “Brain-Inspired SNN for Deep Learning in Time-Space and Deep Knowledge Representation. NeuCube,” in Time-Space, Spiking Neural Networks and Brain-Inspired Artificial Intelligence, N. K. Kasabov, Ed. Berlin, Heidelberg: Springer, 2019, pp. 201–243. doi: 10.1007/978-3-662-57715-8_6
[8] L. Bozhkov, P. Koprinkova-Hristova, P. Georgieva, Reservoir computing for emotion valence discrimination from EEG signals, Neurocomputing, vol. 231, 2017, pp. 28-40, https://doi.org/10.1016/j.neucom.2016.03.108.
Ethics Advisory Board
To assess the ethical questions raised by the project, we are honored to work with two independent renowned philosophers: Doctor Eric Fourneret and Professor Thierry Menissier.
Associate Professor in Philosophy, ETHICS (EA7446),
« Ethics, Technologies and humanities » team at Catholics university of Lille (France)
Eric Fourneret is the author of
Le Cerveau implanté, 2022, Ed. Hermann
Professor at institut de philosophie de Grenoble (France)
Thierry Menissier holds the chair « Ethics and Artificial Intelligence » at MIAI Multidisciplinary Institute in Artificial Intelligence, Grenoble (France)
Events
October 2022 – Kick-off meeting in Grenoble
December 2022 – EIC summit 2022 in Brussels
Results
October 2022 – 1st Place in the Brain-Computer Interface Award 2022 – https://www.bci-award.com/2022
December 2022 – publication of an article with proof of concept of an auto-adaptive ECoG based BMI decoder
https://www.nature.com/articles/s41598-022-25049-w
Rouanne, V., Costecalde, T., Benabid, A. L., & Aksenova, T. (2022). Unsupervised adaptation of an ECoG based brain–computer interface using neural correlates of task performance. Scientific Reports, 12(1), 21316.
NEMO-BMI description of work (deliverable D33)
One medical application of Brain Machine Interfaces (BMIs)s is to translate brain signals into commands for effectors e.g. exoskeleton or electrical stimulation of the spinal cord for motor substitution.
Our application targets severely disabled users, in particular, Spinal Cord Injury (SCI) patients.
SCI leads to severe neurological deficits. Locomotion impairment or the loss of upper limb functions requires caregivers and dramatically impacts the patient’s autonomy in their daily life. Each year, there are approximately 31,000 new cases of SCI in EU and 18,000 in the US, with dramatic human, societal and economical costs. Let us keep in mind that currently there is no approved therapy to improve motor recovery from SCIs. Consequently, there is no BCI system commercially available and the main results published relate to clinical proof of concepts [1], [3].
These revolutionary BMI technologies provides SCI patient with control over orthosis or (neuro)prostheses overcoming some of these disabilities. These recent clinical trials raise great hopes for SCI patients, effectively assessing the feasibility of chronic Electrocorticography (ECoG)-based motor BMI. Wireless minimally invasive ECoG recording systems used in these clinical trials are far more appropriate for long-term application. In terms of effectors, to compensate motor disabilities our consortium supplies patients with a spinal cord stimulation therapy (Targeted Epidural Spinal Stimulation from EPFL) and a dedicated chronic implantable device for spinal cord stimulation (ARCIM from ONWARD).
These clinical trials are very promising but also highlight numerous barriers to bringing Motor BMI to clinical practice or home use. Indeed, brain activity decoding to supply motor functions requires a model in the decoding software. This model is patient specific and has to be trained for each patient in a supervised manner and in a well-controlled environment. For BMI decoding model training, users are requested to perform specific actions: imaging / tentative motor tasks. The neuronal activity is recorded simultaneously. Generally, numerous repetitions are needed to identify the model, which correctly decodes neuronal activity into motor tasks. Machine learning or AI algorithms are generally used for the model training. When the decoding model is stable, it can be used by the patent autonomously.
However, the performances of BCI decoders can degrade over time due to variations in the user’s psychological state, brain plasticity, micro-movements of the brain or other concept drifts. The BMI decoding efficiency degradation requires updating the decoders or training new decoders from scratch, when the performance has degraded excessively. The necessity of regular decoder retraining due to neuronal data instability or variability, is one of the major barriers for long-term use of BMIs at home.
Moreover, targeted epidural spinal cord stimulation can restore movement of paralyzed muscles after spinal cord injury. However, the configuration of the electrical stimulation relies on an almost infinite number of possible combinations of electrodes, stimulation frequency and amplitude. These parameters are currently optimized manually by a team of experts.
In the longer term, the goal is to have devices that are portable, discreet and with a long battery life. To this end, we have started the optimization of the decoding algorithms targeting their integration in a low power application specific integrated circuit (ASIC) (WP5).
All these technological developments are patient centered and embed ethics by design. Their performance will be assessed off-line using existing clinical databases and on-line in new clinical situations after approval of the amendments.
Auto-Adaptive Brain Signal Decoder (WP2)
Facing the constraints of regular Motor BMI retraining, the objective of WP2 is the development of a auto-adaptive BMI (A-BMI) algorithms. After initial BMI decoder training, the BMI decoder will be updated during the BMI free use. The core idea is to infer the training task information directly from the brain signals rather than from the environment, eliminating or reducing the necessity for the special training sessions. The newly proposed Neural Response (NR) decoder will classify the motor actions executed during the BMI free use as ‘correct’ or ‘erroneous’. E.g. if the BMI motor action was considered by the NR decoder as a correct one, with a high level of certainty, this action is added as a label to the corresponding epochs as training data. An adaptive incremental learning algorithm developed at CEA is able to update the current model in real-time using the newly collected data. The A-BMI concept was initially explored off-line to validate the concept.
The objective of the project is (a) to develop A-BMI in more complex and realistic real-life scenarios, (b) to implement A-BMI decoding algorithms, (c) to test them with existing brain recordings, (d) and, finally, to test them in clinical trials, which are already in progress.
Auto-adaptive brain guided electrical stimulation of spinal cord (WP3)
The NEMO-BMI project proposes to use artificial intelligence algorithms to efficiently explore the stimulation space and guarantee the convergence to optimal parameters that can restore movements such as opening the hand, closing the hand, or move the arm in three dimensions.
In particular, the AI algorithm can use a reward function that is based on muscle activity, kinematics, and feedback of the physiotherapist and, of course, feedback of the patient. This feedback from the participant can be measured objectively from the brain signals.
The NEMO-BMI project will develop the tools to measure and decode patient satisfaction and drive the exploration of stimulation patterns to restore the most natural movements after spinal cord injury.
Neuromorphic Approach for Brain Signal Decoding (WP4)
The neuromorphic approach exploits the knowledge about the brain structure and functioning. It consists of two main blocks: a three dimensional spike timing neural network (3D-SNN) and a reservoir of randomly connected firing rate neurons called Echo state network (ESN). The 3D-SNN structure is composed of leaky integrate and fire neurons modelled to mimic the brain neural cells dynamics. Their positions in 3D space were determined according to the personal brain shape of the patient while the connections between them were set initially based on their mutual distance and tuned to reflect the personal brain activity by the brain-inspired spike timing dependent plasticity (STDP) rule. The 3D-SNN structure receives the raw ECoG signals and transforms them to spike trains having time-varying firing rates. The ESN module is a pool of firing rate neurons whose random recurrent connectivity is inspired by the neocortex in the brain. It receives as input firing rates of a group of neurons from the 3D-SNN structure and learns in real time to decode patient’s movement intent. The algorithms are intended to be implemented on a mobile neuromorphic platform.
Miniaturized Hardware for On-Line Decoding (WP5)
The team’s long-term goal is to provide the patient with a solution which is virtually invisible and with a battery autonomy of at least an entire day, if not longer. Currently, the decoding of the brain signals requires a powerful computer, which can extract meaningful features from the raw signals and can then apply a model to decode the features and to identify the patient’s movement intent. Currently, this was done using a high-level compute framework (‘Matlab’) running on a laptop, which the patient carried in a backpack, but this is obviously not an ergonomic solution. The team has already completed a first step of optimization, which consisted of simplifying the algorithm and porting the code to the C++ language. With these optimizations, the decoding algorithm can now operate in real-time on a small, off-the-shelf, portable device. This initial step has resulted in more than a 10X reduction in power consumption. The embedded computer, however, uses a general-purpose processor, which is optimized for running Linux applications, rather than signal processing. The next step which, is already underway, is the development of a dedicated integrated (IC) which is optimized to perform the feature extraction algorithm with minimum power consumption. This circuit will execute the current decoding algorithm, while providing sufficient flexibility to execute new algorithms, which may emerge. In the future, this IC could be integrated in the hardware worn by the patient, or potentially