Science makes it easy to see what you’re thinking

The distinct brain signals that form as thought is translated into speech have helped scientists come up with a solution for those who can’t speak. PHOTO| FOTOSEARCH

When someone is recovering from stroke, they might not be able to articulate their thoughts in words.

Now, neuroengineers have created a system that takes thought from the brain, and translates it into intelligible, recognisable speech for the listener.

When people speak – or even imagine speaking – tell-tale patterns of activity appear in their brain. And when we listen to someone speak or imagine listening, distinct but recognisable patterns of signals emerge.

These patterns have been recorded and decoded in a bid to bring out thoughts hidden in the brain and translate them into verbal speech at will, but achieving this feat has proven challenging.

Early efforts to decode brain signals focused on simple computer models that analysed visual representations of sound frequencies (called spectograms), but this approach failed to produce anything resembling intelligible speech, so researchers turned to a vocoder – a computer algorithm that can synthesise speech after being trained on recordings of people talking.

This is the technology used by devices with voice-controlled intelligent personal assistants, such as Amazon Echo and Apple Siri, to give verbal responses to our questions, said senior author Nima Mesgarani, an associate professor of electrical engineering at Columbia Fu Foundation School of Engineering and Applied Science in the US.

NEURAL PATTERNS

To teach the vocoder to interpret to brain activity, Dr Mesgarani teamed up with a neurosurgeon. They asked epilepsy patients already undergoing brain surgery to listen to sentences spoken by different people, while the researchers measured patterns of brain activity. These neural patterns trained the vocoder.

Next, the researchers asked those same patients to listen to speakers reciting digits between 0 to 9, while recording brain signals that could be run through the vocoder.

The sound produced by the vocoder in response to those signals was analysed and cleaned up by neural networks, a type of artificial intelligence that mimics the structure of neurons in the brain.

The end result was a robotic-sounding voice reciting a sequence of numbers. To test the accuracy of the recording, the team tasked individuals to listen to the recording and report what they heard.

People could understand and repeat the sounds about 75 per cent of the time, “which is well above and beyond any previous attempts,” said Dr Mesgarani.

The improvement in intelligibility was especially evident when comparing the new recordings to the earlier spectrogram-based attempts.

“The sensitive vocoder and powerful neural networks represented the sounds the patients had originally listened to with surprising accuracy.”
The team plans to test more complicated words and sentences, and to run the same tests on brain signals emitted when a person speaks or imagines speaking. Ultimately, they hope their system could be part of an implant, similar to those worn by some epilepsy patients, that translates the wearer’s thoughts directly into words.

“Our voices connect us to our friends, family and the world around us, which is why losing the power of one’s voice due to injury or disease is devastating. With the right technology, voices could be decoded and understood by any listener.

“In this scenario, if the wearer thinks ‘I need a glass of water,’ our system could take the brain signals generated by that thought, and turn them into synthesised, verbal speech,” said Dr Mesgarani.

“This would be a game changer. It would give anyone who has lost their ability to speak, whether through injury or disease, the renewed chance to connect to the world around them.” The findings were published in Scientific Reports. - Science Daily