A MAN with a debilitating neurological disorder is able to ‘speak’ again thanks to a computer that translates brain activity into text that is read aloud.
Scientists have even been able to mimic Casey Harrell’s voice using recordings from before he fell ill.
YouTubeThe joyous moment Casey Harrell could ‘speak’ again[/caption]
YouTubeThe 45-year-old had sensors implanted into his brain[/caption]
YouTubeA computer can read brain signals and turn them into text, which is read aloud[/caption]
The dad, who has sensors implanted into his brain, said: “Not being able to communicate is so frustrating and demoralising.
“It is like you are trapped – often isolated and lonely.
“Something like this technology will help people back into life and society.”
Heartwarming footage shows the 45-year-old with family as he tries the device for the first time.
“He started crying, I started crying, his family started crying,” Dr Sergey Stavisky, a neuroscientist who oversaw the trial, said.
“Everyone had to stop to compose themselves. It was a really special moment.”
Casey was diagnosed with amyotrophic lateral sclerosis (ALS) – a type of motor neurone disease (MND) – in 2020.
He started noticing symptoms after the birth of his daughter.
Initially, doctors suspected he had a muscle-twitching condition that would go away on its own.
Casey then tested positive for Lyme disease, which can mimic the early symptoms of ALS, before neurologists finally realised his limp and cramps were due to the muscle-wasting illness.
His condition continued to deteriorate, and he now uses a specialised chair.
Casey has lost the ability to stand and walk, and can no longer control the muscles used to speak.
But, scientists have developed a new brain-computer interface (BCI) that can interpret his brain signals when he tries to speak.
It then turns them into text that is ‘spoken’ aloud by the machine with 97 per cent accuracy.
It makes people cry who have not heard me in a while
Casey Harrell
Dr David Brandman, a neurosurgeon at UC Davis Health, said: “Our BCI technology helped a man with paralysis to communicate with friends, families and caregivers.
“Our paper demonstrates the most accurate speech neuroprosthesis (device) ever reported.”
The BCI even sounds just like Casey, as researchers used artificial intelligence (AI) to mimic his voice.
“By using video recordings that were collected prior to him having ALS, we were able to digitally reconstruct his voice,” Dr Brandman said.
“So now, when he tries to speak, not only do words appear on his screen, but it sounds like him.”
“It feels a lot like me,” Casey, who is still able to work full time, said.
“It makes people cry who have not heard me in a while.”
YouTubeCasey was diagnosed with amyotrophic lateral sclerosis in 2020[/caption]
YouTubeThe technology worked within minutes for the dad[/caption]
YouTubeFour microelectrode arrays were put into the brain region responsible for coordinating speech[/caption]
What is ALS?
ALS, or amyotrophic lateral sclerosis, is a progressive neurodegenerative disease that affects nerve cells in the brain and spinal cord.
It is the most common form of motor neurone disease (MND).
MND happens when specialist nerve cells in the brain and spinal cord, called motor neurons, stop working properly and die prematurely.
Motor neuron control muscle activity like gripping, walking, speaking, swallowing and breathing.
So as the disease progresses, it can become more difficult to do some or all of these activities.
There is no cure, but treatments can help relieve symptoms. This might include breathing equipment, physiotherapy and speech and language therapy.
ALS is rare, affecting between 1.5 and three people per 100,000 in North America and Europe.
Source: NHS and ALS Association
When Casey first enrolled in the BrainGate clinical trial, he was very hard to understand and needed others to help interpret for him.
In July 2023, he had the BCI implanted. Four microelectrode arrays were put into the left precentral gyrus – a brain region responsible for coordinating speech – to record brain activity from 256 cortical electrodes.
“We’re really detecting their attempt to move their muscles and talk,” Dr Stavisky, an assistant professor in the Department of Neurological Surgery, said.
“We are recording from the part of the brain that’s trying to send these commands to the muscles.
“And we are basically listening into that, and we’re translating those patterns of brain activity into a phoneme — like a syllable or the unit of speech — and then the words they’re trying to say.”
Dr Stavisky added: “It’s not that we’re reading people’s minds, we’re not detecting their inner thoughts, or inner monologue, or even their mind’s voice.
“We’re really detecting their attempt to move their muscles and to talk.”
‘TRANSFORMATIVE TECHNOLOGY’
Previous BCI systems had “frequent word errors”, making it hard for users to be understood.
But within 30 minutes, Casey was using this latest technology to achieve 99.6 per cent word accuracy with a 50-word vocabulary.
“The first time we tried the system, he cried with joy as the words he was trying to say correctly appeared on-screen. We all did,” Dr Stavisky said.
In the second session, the size of the potential vocabulary increased to 125,000 words.
With just an extra 1.4 hours of training data, the BCI achieved a 90.2 per cent word accuracy, which later rose to 97.5 per cent.
“At this point, we can decode what Casey is trying to say correctly about 97 per cent of the time, which is better than many commercially available smartphone applications that try to interpret a person’s voice [about 95 per cent],” Dr Brandman said.
“This technology is transformative because it provides hope for people who want to speak but can’t.
“I hope that technology like this speech BCI will help future patients speak with their family and friends.”
The study reports on 84 data collection sessions over 32 weeks.
In total, Casey used the speech BCI in self-paced conversations for over 248 hours to communicate in person and over video chat.
“It has been immensely rewarding to see Casey regain his ability to speak with his family and friends through this technology,” the study’s lead author, Nicholas Card, postdoctoral scholar in the UC Davis Department of Neurological Surgery, said.
Casey with his personal assistant Emma Alaimo and neuroscientist Sergey StaviskyYouTube
YouTubeCo-directors of the UC Davis Neuroprosthetics Lab – David Brandman and Sergey Stavisky[/caption]
How the technology works
Dr David Brandman, a neurosurgeon at UC Davis Health, said: “Communication is so important to our everyday lives.
“Something as simple as, ‘I have an itch’ or ‘I’m thirsty’ or ‘I love you’ – imagine not being able to say those things to the people that you care about most in your life.
“A brain computer interface is a device that records brain signals for people that are paralysed and then translates those signals to allow people to communicate.
“This research is happening as part of a clinical trial called BrainGate2.
“The idea is that people living with conditions like spinal cord injury, brain stem stroke or ALS undergo surgical implantation of electrodes into their motor cortex.
“And thereafter, they work with a scientific team to develop next generation technologies to help people living with paralysis.
“I implanted Casey Harrell, 45, who has ALS, who really can’t be understood when he tries to speak very easily.
“By using a BCI, we can translate his desire to speak and have words appear on the screen.”
Dr Sergey Stavisky, a neuroscientist, added: “In essence, what we’re doing is bypassing the injury.
“We are recording from the source, from this part of the brain that’s trying to send these commands to the muscles, and we’re translating those patterns of brain activity into the phonemes – like a syllable, the unit of speech – and then the words that they’re trying to say.”
Dr Nick Card, postdoctoral scholar and lead author, said: “Not only can we accurately decode the words that he’s reading prompted off a screen, but we can also accurately predict what he’s trying to say in freeform conversation.”
Dr Stavisky said: “It’s not that we’re reading people’s minds, we’re not detecting their inner thoughts, or inner monologue, or even their mind’s voice.
“We’re really detecting their attempt to move their muscles and to talk.”