Woman with severe paralysis to communicate with brain-computer interface

UC San Francisco and UC Berkeley researchers have achieved a groundbreaking development in brain-computer interface (BCI) technology, enabling a woman severely paralyzed by a brainstem stroke to communicate through a digital avatar. This marks the first instance of synthesizing both speech and facial expressions from brain signals, showcasing remarkable progress beyond existing commercial technologies. The findings, published in Nature on August 23, 2023, present hope for the future approval of an FDA-sanctioned system facilitating speech from brain signals.

Dr. Edward Chang, Chair of Neurological Surgery at UCSF, and his team, who have dedicated over a decade to BCI research, have surpassed their previous milestone of decoding brain signals into text. This new achievement involves decoding the richness of speech and the accompanying facial movements that bring conversation to life. Using a paper-thin rectangle of 253 electrodes implanted onto critical speech-related areas of the woman's brain, the researchers intercepted signals intended for her muscles—those responsible for speech and facial expressions.

A cable connected the electrodes to a computer bank, and over weeks of collaboration, the participant trained the artificial intelligence algorithms to recognize her unique brain signals for speech. The training process included repetitively speaking various phrases from a 1,024-word conversational vocabulary. Notably, the team adopted a novel approach, decoding words from phonemes—sub-units of speech, akin to letters forming written words. By training the computer to recognize 39 phonemes, the system became not only more accurate but also three times faster. The researchers brought the digital avatar to life using software from Speech Graphics, a company specializing in AI-driven facial animation. Customized machine-learning processes allowed the software to synchronize with the woman's brain signals, translating them into the avatar's facial movements, including jaw motion, lip protrusion, tongue activity, and expressions of happiness, sadness, and surprise. Looking ahead, the team aims to develop a wireless version of the BCI, eliminating the need for physical connections. This step forward holds promise for individuals with severe paralysis, offering a potential avenue for them to regain the ability to communicate and express themselves through advanced BCI technology.