Accessible Music and Creative Code

The music technology program at NYU Steinhardt represented an ideal interconnection between Payne’s ranging interests in music composition, computer science, and accessibility.

“FMDG emerged as an amazing partner as I was working on my dissertation about the nexus of creativity, accessibility, and learning,” says Payne, who is now an assistant professor of user experience and design at the University of North Carolina Chapel Hill’s School of Information and Library Science.

Payne’s first projects included transcribing printed scores to help create the braille music read by FMDG students and interviewing blind and low-vision musicians across the country. He and his advisor, Amy Hurst, associate professor of occupational therapy and director of The Ability Project, co-authored a paper on the challenges and design opportunities for providing access to music notation in braille and large print formats.

“We discovered a bunch of innovations that vastly simplify the process of creating braille music compared to even 20 years ago,” says Payne. “However, music notation is still thought to be a visual medium, and significant labor is required to take a single print score and accurately convert it to braille – not to mention a library of print scores.”

This scholarly endeavor led to Payne developing a tool called SoundCells with a team of FMDG technologists and NYU undergraduate research assistants. SoundCells allows people to write music that is automatically created accessible, with both braille and print notation generated simultaneously. The work culminated in an online six-week class at FMDG for blind and low-vision musicians who learned how to compose short pieces using the software. The class then performed their works at a Zoom concert. Payne’s first article on SoundCells won the Best Paper Award at the Web4All Conference.

The current iteration of Payne’s collaboration with the school is FMDG’s FiLOrk (Fil’ Laptop Orchestra), an electronic music ensemble led by high school students who identify as blind or low vision writing code to joint-compose and perform original music. FiLOrk had its first concert in December of 2022 and several more throughout 2023. 

“Laptop orchestras have been around for a while, but mainly at universities,” says Payne. “Our challenge here is to make the curriculum friendlier to younger learners and the classroom more inclusive. Since they are responsible for writing the score and building the instruments with code, everyone wears a lot of hats. Each decides what role they want to take in a performance in pursuit of a collective musical vision. It’s a lot of fun and at times a bit chaotic!”

Payne still collaborates with FMDG as a vital research partner; FiLOrk is currently supported by a Project Grant from the Consumer Technology Association (CTA) Foundation. In addition, the NYU undergraduates who worked with him and FMDG during his fellowship continue their research from New York while Payne contributes remotely from North Carolina.