Welcome to the Holodeck

NYU Steinhardt News

Welcome to the Holodeck

Holodeck illustrationFor years a group of computer geeks, inventors, game theory mavens, engineers, music tech nerds, math whizzes, informaticians, and other digiterati across the NYU spectrum collaborated from time to time on various projects pegged to their mutual fascination with virtual reality (VR). What they dreamed of (but couldn’t afford) was the chance to build a dazzling, futuristic supercomputing environment where all of their projects could interact and exponentially grow—something like the Holodecks on board Star Trek starships ... but in the heart of New York City, not science-fiction outer space.

In late 2016, on their second try, this group won a $2.9 million grant from the National Science Foundation (NSF) to do just that. The university is contributing another $1.2 million to the project.

Fans of sci-fi probably remember that the Holodeck often functioned as a psychological Club Med for earthling crew members who were homesick for their culture. They would whoosh through the octagonal door to the Holosuite and, depending on the program they selected, could soon be tramping through the forests of their home planet, exploring the Wild West, playing on rival virtual baseball teams, or hanging out with Sherlock Holmes. These dreamscapes were said to use an advanced technology in which holographic people, places, and things seemed as solid as the actual physical world.

NYU doesn’t have a Holosuite—it has three, all with normal doors. Two are in Manhattan, one at the Nursing Informatics Lab on First Avenue and the other at the Media Research Lab off Washington Square, and a third in Brooklyn at the Media and Games Network. The applications of the NYU Holodeck project are spread among four schools: the Courant Institute of Mathematical Sciences; the Rory Meyers College of Nursing; the Steinhardt School of Culture, Education, and Human Development; and the Tandon School of Engineering.

The NYU group has specialized in programs that let multiple users experience the Holodeck together in real time: think classes, conferences, and concerts rather than preprogrammed video games. “It’s all about looking at the future of conversation,” says Holodeck collaborator R. Luke DuBois, associate professor of integrated media technology at Tandon. “If you go back to those cheesy ads from the ’80s—‘Reach out and touch someone’—that’s really the premise of how to bring people together when they’re in different places. If you’re in San Diego and I’m in Brooklyn, can we draw something together, or perform a musical piece together, or explore a data set together if we’re medical researchers?”

Holodeck illustrationDespite its name, the NYU Holodeck isn’t grounded in holography—no Hogwarts bathroom ghosts or Princess Leia “Help me, Obi-Wan Kenobi” messages, much less the full-immersion starship technology. Although the five-year project is only in its initial stages, the programs under way usually require participants to wear goggles or headsets.

The goal, says Jan Plass, professor of digital media and learning sciences at Steinhardt, is “playful learning.” So, in other words, you might not be able to solve cases in an alternate universe with Holmes and Watson, but the educational stuff will still be entertaining. In the following stories are what some of the collaborators from the four schools have in mind for the future, separately and together.

Steinhardt School of Culture, Education, and Human Development
Agnieszka Roginska
Music Associate Professor of Music and Music Education

Sound technology is a crucial component of virtual reality. “If you have a visual environment where you are inside a teapot, but it doesn’t sound like you’re inside a teapot, it’s not convincing,” Roginska notes. Moreover, the visuals and the sounds have to be synchronized—“not just what you hear, but how when you move, the sound has to follow you in a very specific way,” she adds.

For instance, in [an] Alice in Wonderland theater piece, a graduate student Roginska advises, Marta Olko, designed soundscapes to accompany the magic realism visuals. When a virtual bird flew through the scene, it had to sound to all the participants’ headsets the way a real bird flapping and cawing would sound at all the different locations in the room. When everyone in the room seemed to shrink in size, the sounds—the rain, the crickets, everything else—had to change, too.

The goal within the Holodeck is “to get to the point where you can’t differentiate a real sound from a virtual sound ... which is a challenging thing to do,” says Roginska. Some advances will come externally, as headsets and other gear improve. Others will result from programmers figuring out (with software, recordings, and various algorithms) how best to re-create what Roginska calls each sound’s “acoustic fingerprint”—a matter of where the sound happens as well as what it is. A person talking or playing an instrument inside a cathedral sounds different than the same sound in a park or in a living room, although much of this auditory knowledge happens at an instinctual level. “Our brains have evolved over thousands of years to pick apart information from the sound, because that gives us clues to where we are and if there is any danger coming,” Roginska says. “When we go into the virtual environment, it’s exactly the same thing.”

The Holodeck holds out the promise of making any group whose dealings would be enhanced by face-to-face contact truly feel as if they are in the same, say, orchestra pit and to reap the benefits of that feeling of connectivity. Roginska’s students are researching what happens when two musicians are playing in the same room versus when they’re performing together in separate studios. “There are significant differences in the performance, the phrasing, the tempo, and other factors,” Roginska reports. At this point no one knows exactly why togetherness encourages creativity, she adds, but “something magical happens when you are in the same space. On the Holodeck, what we want to do is create that space as much as possible and make it indistinguishable both in the sound but also in how people interact.”

By Lindsy Van Gelder
Illustrations by Jun Cen
This article originally appeared in the NYU Alumni Magazine