Crowdsourcing Can Help Researchers Assess Efficacy of Parkinson’s Treatment, Research Finds

While testing a voice treatment for people with Parkinson’s, researchers found crowdsourcing a reliable method of evaluation.

Untrained listeners who are crowdsourced to assess voice treatment for people with Parkinson’s can reach conclusions that agree with reliable measurement tools, finds a team of speech-language pathology researchers.

“Studies of speech and voice treatment methods for individuals with Parkinson’s Disease typically evaluate progress using fine-grained acoustic measures or highly trained listeners’ perceptual ratings,” says lead author Tara McAllister, NYU Steinhardt Associate Professor. “However, this approach leaves some uncertainty about the real-world impact of these changes, since it is not always clear they would be apparent to untrained listeners that patients interact with in their daily life.”

Images of interconnected avatars on map of the US. — Gettyimages/metamorworks

In a study published by the Journal of Speech, Language, and Hearing Research, McAllister and her co-authors successfully replicated a 2022 study (Moya-Galé et al.) in the Journal of Voice that showed improved voice quality for people with Parkinson’s after receiving the Lee Silverman Voice Treatment (LSVT LOUD). The results of the original study were determined using the Acoustic Voice Quality Index (AVQI), a metric that combines multiple acoustic measures of speech to arrive at a single value that correlates highly with expert listeners’ ratings.

In crowdsourcing, researchers use platforms such as Amazon’s Mechanical Turk to connect with a large number of individuals who perform tasks online for monetary compensation. The researchers in this study used an initial task to determine the reliability of crowdsourced listeners in assessing audio samples. Thirty eligible raters were then asked to evaluate the 250 voice recordings from the Moya-Galé et al. study, which were divided into blocks of 50 recordings each (25-27 listeners for each block).

We think that crowdsourcing has exciting potential to make it easier for researchers to understand the real-world relevance of the treatment methods they study, and ultimately support better alignment of priorities between researchers and patients.

Tara McAllister, NYU Steinhardt Associate Professor, et al.

The authors found a statistically significant correlation between the listeners’ ratings and the AVQI scores. They also replicated the original study’s finding that significant gains in voice quality were observed after LSVT LOUD treatment but not after two comparison treatments.

“Overall, these results support the use of crowdsourcing to evaluate speech samples collected from clinical populations, even for constructs such as voice quality that may be less familiar to untrained listeners,” they write.

“We think that crowdsourcing has exciting potential to make it easier for researchers to understand the real-world relevance of the treatment methods they study, and ultimately support better alignment of priorities between researchers and patients.”

In addition to McAllister, the study’s authors include Christopher Nightingale, a doctoral student at Gallaudet University; Gemma Moya-Galé, an assistant professor at Long Island University; Ava Kawamura, an undergraduate student at Georgetown University; and Lorraine Olson Ramig, a professor emerita at the University of Colorado at Boulder and chief scientific officer at LSVT Global, Inc.

This study was funded by grants from the National Institute on Deafness and Other Communication Disorders (R01DC001150 and R01DC017476) and a contribution by LSVT Global, Inc.

Editor’s note: Lorraine Olson Ramig is employed as the chief scientific officer and has ownership interest in the for-profit company LSVT Global, Inc. Her conflict-of-interest management plan has been approved by the Office of Conflict of Interest and Commitment at the University of Colorado.