A groundbreaking medical technology has empowered a severely paralyzed woman to communicate through a digital avatar by converting her brain signals into speech and facial expressions. This achievement offers hope to individuals who've lost their ability to communicate due to conditions like strokes and ALS.
Previously, patients had to rely on slow speech synthesisers, often involving eye tracking or subtle facial movements to construct words, making it difficult to conduct natural conversations. According to a video published by UC San Francisco (UCSF) on YouTube, the new technology utilizes minuscule electrodes on the brain's surface, detecting electrical activity in the brain areas controlling speech and facial expressions. These signals are instantly translated into speech and corresponding facial emotions by a digital avatar, enabling expressions like smiles, frowns, or surprise.
The patient, a 47-year-old named Ann, suffered severe paralysis for over 18 years following a brainstem stroke, rendering her unable to speak or type. Typically, she communicated slowly, at up to 14 words per minute, using motion-tracking technology. Ann now aspires to work as a counsellor with the help of the latest avatar technology.
The research team implanted 253 paper-thin electrodes on Ann's brain surface, specifically over speech-related regions. These electrodes intercepted the brain signals that would have controlled tongue, jaw, larynx, and facial muscles had it not been for the stroke.
Following implantation, Ann collaborated with the team to train an AI algorithm to recognize her unique brain signals for different speech sounds. The computer successfully learned 39 distinct sounds, and a ChatGPT-style language model converted these signals into coherent sentences. These sentences were used to control an avatar, resembling Ann's pre-injury voice based on a recording from her wedding.
While not flawless, with a 28 percent of word decoding error rate and a brain-to-text rate of 78 words per minute (compared to the typical 110-150 words spoken in natural conversation), these advances indicate practical utility for patients.
Professor Edward Chang, leading the initiative at the University of California, San Francisco (UCSF), stated, "Our goal is to restore a full, embodied way of communicating, which is really the most natural way for us to talk with others. These advancements bring us much closer to making this a real solution for patients."