AI Can Now Detect Depression From Your Voice, And It’s Twice As Accurate As Human Practitioners

When children feel unwell, the mother often senses something is amiss just by looking at them. She springs into action immediately. By diagnosing the problem, the mother uses a mix of home remedies, nutrition, activity, and rest to nurse her kids back to good health. 

Timely care and motherly attention fix the health issue long before it develops into an ailment. In most cases, the children may not realize they were unwell.

“Wouldn’t it be great if we could detect and treat symptoms of depression in adults in the early stages, just the way mothers sense health challenges with their kids?” asks David Liu, CEO of Sonde Health.

Today, mental health poses one of the biggest challenges in the world. The US Center for Disease Control and Prevention (CDC) is concerned that one in two Americans could suffer from depression post-pandemic. Medical estimates indicate that around two-thirds of all depression cases go undiagnosed. 

Thanks to recent breakthroughs in artificial intelligence (AI), providers can now detect depression just by listening to someone speak a few sentences. Surprisingly, the language or the words spoken aren’t as important as how you say it. 

Let’s find out how to improve mental health with AI and explore significant trends emerging in this space. We’ll see how some promising companies are innovating with AI to provide motherly attention for mental healthcare.

Twin challenges in mental health today

Mental healthcare faces two main issues. Firstly, it is not easy to get access to mental health professionals on a timely basis. Secondly, for patients who manage to get professional help, the diagnostic process and quality of care are not consistent.

Rima Seiilova-Olson moved to San Francisco as a research analyst in 2015. A few years later, when she gave birth to her first child, she experienced postpartum “baby blues.” Most mothers experience mood swings in the first few weeks after delivery. But for some, this develops into a more severe condition called postpartum depression.

Seiilova-Olson decided to seek professional help. She remembers making frantic calls to her healthcare provider for appointments. After many attempts, she managed to schedule a visit with a therapist. The appointment was more than two months away. She had no option but to wait.

Around this time, Seiilova-Olson participated in the Open AI Hackathon organized in the Bay Area. There, she met Grace Chang, a technologist, and they bonded immediately. Apart from being among the few women participating in the technology event, they shared frustrating personal experiences in getting help for mental health. Chang and Seiilova-Olson soon founded Kintsugi, a startup that uses AI to democratize access to mental health.

The mental health system has a clear capacity constraint due to a scarcity of licensed health professionals. For every ten individuals who may suffer from mental health issues, only three secure access to the system. “To make matters worse, there aren’t efficient processes to use this scarce bandwidth,” says Chang. “Many critical cases go waiting while less severe ones end up using precious bandwidth.” This is the first stumbling block facing millions of patients.

When patients brave these logistical nightmares to get professional help, they run into the second critical challenge: care quality. Today, the diagnosis of mental health issues is based on screening tools such as the Patient Health Questionnaire (PHQ). 

“The challenge is that these questions aren’t very objective and are heavily dependent on what the patient can remember from the past few weeks,” says Chang. The doctor’s diagnosis then rests on the accuracy of the picture recreated from a mental health patient’s memory. 

Not surprisingly, only 47.3% of mental health cases are detected accurately by professionals. Imagine sending home one out of every two patients, assuring them that they are alright, only to discover that their problems have worsened or become life-threatening.

Rebooting mental health with artificial intelligence

“Digital health solutions could address the twin issues of access to timely care and consistent quality of care,” says Shwen Gwee, VP and Head of Digital Strategy at Bristol Myers Squibb. He sees great promise in the rise of digital therapeutics, which are evidence-based, clinically evaluated software tools that help treat, manage, and prevent a broad range of diseases.

Can these innovations diagnose complex health conditions just from the human voice?

“When we listen to a person speaking, we notice variations in pitch, energy, tonal quality, and rhythm,” says David Liu, CEO of Sonde Health. “By processing this audio, we can break down a few seconds of voice recording into a signal with thousands of unique characteristics.” This method is called audio signal processing.

What’s less known is that small changes in someone’s voice every few milliseconds can result from changes in their body and health conditions. With this rich data, it is possible to identify which vocal features map to particular disease symptoms or changes in health. Using data from thousands of individuals who suffer from certain health conditions, we can teach AI algorithms to detect the vocal patterns that are common among these patients.

Liu adds, “Once we confirm that these parameters are good indicators of a particular health condition symptom, we refer to this specific subset of acoustic features as ‘vocal biomarkers.’ We then carefully evaluate which of these biomarkers work best in different settings, across diverse groups and patient demographics.”

The team at Sonde used this approach to train machine learning (ML) models that can provide cues when people start experiencing depressive symptoms. This model uses six vocal biomarkers that measure aspects such as how well you can hold your vocal pitch or how dynamic your voice is when speaking.

While the approach sounds promising, there’s a key challenge in building these AI solutions. It’s not easy to get high volumes of good-quality data (i.e., historical voice samples from patients tagged in a way to train the algorithm). Sonde acquired its voice data through research studies, partnerships, and crowd-sourcing. They acquired NeuroLex Labs, a company that provides online, voice-based surveys. With this multi-pronged approach, Sonde’s repository has over 1 million voice samples from over 80,000 people globally.

When you have patient voice data from several countries, should you create many versions of these AI models? “Not really,” says Seiilova-Olson of Kintsugi. Her team shared an interesting finding from their research. “We discovered that our AI model’s results did not depend on the language the people were speaking, their age or gender, or even the part of the world they lived in.” 

The voice biomarkers accurately flagged symptoms that transcended all such differences. With just 20 seconds of an audio clip, Kintsugi’s AI solution detects mental health issues with over 80% clinical accuracy. Compared to practitioners' current rate of detection (47.3%, as noted earlier), artificial intelligence can almost double the effectiveness of patient diagnoses.

Clearly, digital therapeutics and voice biomarkers can help save lives.

How AI improves patient care, one conversation at a time

Kintsugi and Sonde offer their AI solutions as consumer apps that can continuously monitor individual health. Additionally, their AI is integrated into channels such as telehealth platforms and care management applications to aid clinicians.

Thus, AI gets into action even before a patient may sense the need for intervention. “Think of this as a fitness tracker for your mind,” says Liu. For example, Kintsugi’s consumer app has a voice journaling feature that monitors patient health daily.

By being continuously available for patients, AI precludes the need to schedule an appointment. By accurately pre-screening patients, it saves precious bandwidth in the mental health system. That’s how AI solves the first challenge of access to timely care. 

When patients do come in for appointments, practitioners use AI solutions to augment their diagnoses. “The typical time between patient visits could be anywhere from weeks to years,” says Gwee. “What happens in between is usually lost to physicians. Thanks to continuous digital monitoring, physicians now have access to fine-grained data that is far more reliable.”

When clinicians have a live conversation with patients, the AI identifies vocal cues of clinical depression and anxiety in real-time. It then lets practitioners know whether they should plan a follow-on consultation. This AI-driven tracking and targeted intervention can save lives.

This way, AI addresses the second challenge of effective diagnosis and consistent quality of mental healthcare. For example, Cognitive Behavior Institute is a mental health practice in Pennsylvania that treats folks with depression, anxiety, and substance abuse. A long-term patient was struggling with anxiety but couldn’t pinpoint triggers of her anxiety episodes. 

She started using Sonde’s Mental Fitness App for two weeks as part of a pilot. The results from her voice samples helped identify specific thoughts and events that led to her symptoms. She was surprised that some of these triggers she was alerted about weren’t obvious.

Her clinician received the data in the form of health scores on a dashboard. It enabled the therapist to check in and intervene, particularly when the patient’s perceptions differed from what the app said. 

“AI can transform mental health; however, we must watch out for some risks when they are deployed in the real world,” says Sathiyan Kutty, Head of Predictive Analytics at one of the largest healthcare organizations in the US. AI is as good as the data from which it learns. Kutty notes that AI solutions can be biased because the data often come from people experiencing mental health struggles rather than those who are healthy. Mitigating this risk calls for balancing data samples with enough healthy individuals.

“Organizations often get carried away by the reported effectiveness of AI solutions,” says Kutty. That is, they design AI to make critical decisions without keeping human practitioners in the loop, or they fail to plan handovers to clinicians at appropriate stages. “AI should be used to augment the capabilities of physicians and not replace them,” he adds.

Hey Alexa, how am I doing?

Today, the momentum for AI solutions in mental health is building rapidly. “A year ago, I would have to explain what vocal biomarkers are,” says Liu. “Today, no explanation is necessary; instead, we are discussing specific use cases for our technology.”

With the explosion of smart devices, access to voice data is becoming ubiquitous. “Whether it's your mobile phone, wearable, or personal assistants like Alexa, the collection of passive data is seamless and already part of our lifestyle,” says Gwee.

“We are starting to see examples of AI-driven biosensors that can predict heart failure days in advance of an actual event. That's the power of passive (real-world) data collection, combined with patented algorithms and machine learning. Similarly, digital therapeutics platforms for mental health can predict when somebody is falling into a depressive episode without waiting for the critical event to occur,” Gwee adds.

Then, it’s probably no exaggeration to say that artificial intelligence is at the advent of providing motherly attention and preventative care for mental health.