NYU Funds Interdisciplinary Research on AI Tools for Speech Therapy

The funding will support creating AI-driven tools that offer targeted feedback for clinicians to better support speech therapy for children.

Tara McAllister, associate professor of communicative sciences and disorders at NYU Steinhardt and director of the doctoral program, has been awarded an NYU Discovery Research Fund for Human Health grant to create AI-driven tools that offer targeted feedback for clinicians to better support speech therapy for children.

While McAllister is the Principal Investigator (PI) on this project, she is working with several co-PIs:

• Amanda Eads, PhD candidate in communicative sciences and disorders at Steinhardt
• Brian McFee, assistant professor of music technology and data science, NYU Steinhardt and Center for Data Science
• Hai Shu, assistant professor of biostatistics, NYU School of Global Public Health (GPH)
• Taehyo Kim, PhD student in biostatistics at GPH

The team’s work aims to address the shortage of trained speech pathologists for children in need of speech therapy by developing tools that can make clinicians more effective and also support home practice.

“Technology currently being used in clinical speech therapy is built on speech recognition technology that is mostly just able to determine whether a patient has said a word right or wrong,” says McAllister. “It doesn’t offer qualitative feedback—such as ‘you were close to the right pronunciation but try moving your tongue in this way instead’—and that’s the gap we’re trying to bridge with this work.”

“This research is multimodal, taking the existing technology and adding both audio and ultrasound data to get at this problem of adding more qualitative feedback,” says Eads, on whose dissertation work this project is based. “I started my doctoral journey looking at ultrasound images from children producing the ‘r’ [or rhotic] sound to examine their tongue shapes. We’re collecting lots more of these ultrasound images from children aged 9 through 17 across New York and New Jersey.”

The ultrasound images will also be paired with audio recordings of the children’s pronunciations so they can pair up what is happening with tongue shapes for both perceptually accurate and inaccurate rhotic productions.

“We’ll be using machine learning for audio signal processing to look for and label patterns,” says McFee. “We’ll be able to map the sound to the image, which then corresponds to this kind of mistake the patient is making and automatically suggest a corrective measure to the clinician.”

One of the distinctive elements of the research team’s composition is its interdisciplinary nature.

“This is fundamentally a speech problem, but our collaborators at GPH are experts in medical imaging—like brain tumors and breast cancer—and Brian is fundamentally a music researcher,” says McAllister. “It’s a very ‘Steinhardt’ approach to bring in these people with diverse perspectives to see how we can work together to create something new and effective.”

The team’s goal by the end of the year is to have two highly accurate classifiers for perceptually accurate and inaccurate rhotic productions; next, they can determine the best way to leverage the tool they’ve created to help clinicians currently, as well as think about longer-term uses in clinical and even at-home care.

In one potential model, a child would work with a speech pathologist on their rhotic pronunciation in a clinical setting once or twice a week, and then be able to go home and practice on their own with gamified software that keeps them engaged and gives them qualitative feedback just like their clinician would.

While the research draws clinical data from one of McAllister’s previous grants from the National Institutes of Health, this study is actively recruiting children ages 9 through 17 with and without rhotic pronunciation difficulty. Learn more and enroll a child to participate at a lab in NYU Washington Square; Montclair, NJ; or Syracuse, NY:

• Children with typical “r” pronunciation
• Children with difficulty producing “r” sound