Illustrations by Julia Yellow, courtesy of Scope, Issue 6, 2025
By observing babies at home for hundreds of hours, researchers are gaining unique insights into child development and the ways little ones learn. Turns out the tots also have something to teach AI.
Researchers Karen Adolph and Catherine Tamis-LeMonda study seemingly disparate areas of child development: Adolph, a professor of psychology and neural science at NYU College of Arts and Science, focuses on motor skills and runs NYU’s Infant Action Lab, while Tamis-LeMonda, a professor of applied psychology at NYU Steinhardt, is an expert in language acquisition who heads up NYU Steinhardt’s Play and Language Lab.
Still, the women have been close collaborators for more than two decades, drawn together, in part, by a mutual frustration with the way that human development is typically studied. They discovered this shared viewpoint almost serendipitously. “One day, I said, ‘Karen, I want you to read this new paper I wrote about everyday language in the home environment versus the lab,’” remembers Tamis-LeMonda. “And she said, ‘Oh my god! I have the same paper, but it’s about walking in the home versus the lab!’ What we both were really struck by was that so many of our models and theories about how children learn are based on data from very brief prestructured lab tasks, but if you look at a child’s speech or how they walk or play at home, it’s quite different. In the lab we say, here’s the toy you’re going to be playing with. But as any parent can tell you, if you watch kids play at home, you realize they’re often more interested in digging around in the houseplants and messing with the remote control.”
Illustrations by Julia Yellow, courtesy of Scope, Issue 6, 2025
Indeed, some of developmental science’s most enduring theories stemmed from observations outside of a laboratory. Attachment theory, for instance, grew out of watching hospitalized babies when they were separated from their mothers, and Jean Piaget’s most famous work was based on his own three children. “And so we had this idea,” says Tamis-LeMonda. “Why don’t we start looking at children in their natural environments?”
That idea was the initial spark for a hugely ambitious project known as PLAY—an acronym for Play and Learning Across a Year—for which researchers at 30 sites across the United States have collected hundreds of hours of video capturing babies aged 12 to 24 months in their own homes. The videos are all transcribed, time stamped, and annotated, with investigators—some of whom are NYU undergraduates—writing down every giggle, babble, smile, or shout. “If you’re studying babies crying, for instance, you can search through thousands of hours of videos to find instances where babies are crying,” says Adolph. “You can then see, for example, how did the mom react? What prompted the baby to stop crying?”
For Adolph and Tamis-LeMonda, PLAY has consumed the better part of a decade, and, from the beginning, they were thinking big. Their goal was to create a large, diverse dataset—which is the first of its kind—collected in accordance with highly detailed, mutually agreed-upon protocols. Their vision: that the work could be used not only for their own research but also by scientists around the world studying many different aspects of child development.
To achieve that goal, they knew they needed the participation of experts in a wide variety of domains. And so, in 2016, they began reaching out to their peers at NYU and at other research institutions across the country—some who studied play, for example, and others who focused on learning, fine motor skills, or data sampling. Once collaborators were on board, they met over Zoom to iron out exactly how the videos would be collected and annotated—a process that involved many hours of debate. The entire group then convened on the campus of the National Institutes of Health to finally arrive at their shared methodology.
Illustrations by Julia Yellow, courtesy of Scope, Issue 6, 2025
Initially, Adolph and Tamis-LeMonda thought they’d attempt to capture eight hours of video about each baby, but after running a pilot program themselves, “we quickly learned that for two to four of those hours, you’re in the dark watching a baby sleep,” says Tamis-LeMonda. After their early observations determined that babies’ behaviors are relatively stable from one day to the next and from hour to hour, they ultimately decided that “all you need is one good hour,” says Tamis-LeMonda.
In 2019, PLAY was finally ready to launch—only to be waylaid by the pandemic. But by 2021, researchers were able to safely enter the subjects’ homes. In the time since, they’ve collected 900 hours of video, working with 900 families across the United States. “We have English and Spanish speakers, a range of socioeconomic status households, and homes that range from cramped New York City apartments to big houses in Iowa,” says Tamis-LeMonda. “We’ll be able to look at all of these different factors and how they’re playing into babies’ behaviors.”
The data annotation process is ongoing, but in the meantime researchers have access to videos from their own local sites and are already putting them to use. One group is digging into the effects of screen time, for example, and another is looking at how a parent’s use of gestures relates to their child’s language development.
Among the findings that are starting to emerge: infants are two to three times more likely to produce words for objects that they manipulate and that their mother names—for example, if a baby picks up a sock and their mother says, “Sock”—than for objects they don’t manipulate and their mother doesn’t name. Additionally, their data supports the aforementioned observation that kids often prefer everyday objects over toys, finding that even in homes filled with toys, infants spend just as much time playing with things like boxes, cups, cabinets, and drawers as, say, cradling their dolls or pushing toy cars around the rug. Adolph is using the PLAY data to study how the size of a home and the surfaces inside it affect walking and crawling. “I’m asking, ‘Does it matter if the home is big or small?’” she says. “Do kids in tiny apartments just go around and around and around and get as many steps doing laps around the table versus navigating a large environment?”
So many of our models and theories about how children learn are based on data from very brief prestructured lab tasks, but if you look at a child’s speech or how they walk or play at home, it’s quite different.
Once all of the sites have completed work, there will be an embargo period during which only the PLAY researchers will have access to the full dataset. “And after a year, once those people have published lots of papers—and lots of papers will come out of it—we’ll be able to hit a button and share it with 2,000 investigators across the globe,” says Adolph. They’ll do that with Databrary, a digital library that Adolph established in 2012 for sharing videos with other developmental behavior researchers. Determined to keep their work as transparent as possible, Adolph and Tamis-LeMonda will also upload all of those many Zoom meetings and discussions among collaborators so other scientists can learn about their methods and protocols and discover exactly how and why they were established.
And soon, they hope, even more collaborators will be putting those methods to use, as Tamis-LeMonda and Adolph envision taking PLAY global. They’re already launching sites in China, Hong Kong, South Korea, and Tajikistan. “If you look at the homes we’re visiting here, there are books and shape sorters and all sorts of child-designed toys,” says Tamis-LeMonda. “That’s going to look a lot different if you go to rural Ghana or Kenya. We’re really trying to understand how children play and learn in all different types of environments.”
Adolph also sees applications for PLAY beyond developmental science. “It will be an unprecedented dataset for AI training models,” she points out. “Right now, to teach AI systems object recognition, they’re using videos off YouTube, and to teach speech they’re using things like news broadcasts—that’s not natural language. To be able to refine AI systems with videos of real kids in real environments is really critical to machine learning.”
One computational cognitive scientist at NYU is already on it. For his study, Brenden Lake, associate professor of psychology and data science at NYU Courant Institute of Mathematical Sciences, posed the question: Could an AI system learn more like a human baby does? Currently, he explains, the way AI systems learn language and the way children learn to talk is vastly different. Babies, in short, appear to be much better learners. Whereas the most sophisticated AI systems, like ChatGPT, are trained using astronomical amounts of data with a word count in the trillions, children learn language by exposure to a word count that’s only in the millions.
“It would take a child about 100,000 years of listening to their parents to get exposure to the number of words an AI system is trained on,” Lake says. Various possible explanations can account for this difference, including the idea that AI systems learn in a fundamentally different way than humans, requiring exponentially more input. To test that idea, Lake set out to train a state-of-the-art AI model, not on everything available on the web but on input from the eyes and ears of a single child, a boy named Sam, who wore a light head-mounted camera for approximately two hours each week from the age of 6 months to 2 years to capture footage of his natural experience at home. Published in the journal Science, Lake’s findings showed for the first time that an AI neural network trained on developmental input from a toddler can learn language, even with such a small quantity of information.
While the toddler-trained AI was not able to speak as fluently as a system like ChatGPT—“it’s rather limited in comparison,” says Lake—it could, for example, recognize a ball and learn what “sand” means. “It did start to make real connections between spoken language and visual objects,” he says. And these findings not only offer information about how AI systems learn; they might also tell us something about human learning.
“For decades, cognitive science has asked the question, ‘What ingredients do children need to learn words?’ ” says Lake. “How does a child come to recognize that ‘ball’ means a round, lightweight object that you can play with, when every time they hear the word ‘ball’ there are many other objects the word could refer to? ‘Ball’ could refer to the carpet in the playroom, or even the time of day or the smell of the room at that moment.” Over the years, researchers have proposed dozens of theories about how children are able to make these object-word connections, including the hypotheses that shape is more important than other properties and that children pay attention to social cues from their parents. “But then we trained an AI model that doesn’t have any of these supposed human cognitive biases or any help from social cues and found that it begins to map words to objects anyway,” says Lake. “This might tell us that these biases aren’t totally essential in order to achieve some basic language acquisition.”
Lake is quick to point out that this round of research is only the beginning when it comes to digging into these essential questions. “We trained the system on only about 1 percent of what the child would have seen in a two-year period,” he says. “The next step would be to understand what would be possible if more data was available. If we could assess what prevents today’s AI technology from really learning and developing like a child does, it might explain the gap between human intelligence and development and AI intelligence and development.”
In order to get that data, the study is continuing to a next phase—and this time Lake is involved on an even deeper level: his own daughter, Luna, is one of the subjects. Since the age of 6 months, she’s been wearing a helmet cam, providing video data of the world from her own tiny point of view. It’s been gratifying to know, says Lake, that his family is making an important contribution to science, and fascinating to see the world through Luna’s eyes. Still, relying on one’s own toddler for scientific data comes with some challenges. “Initially, she was happy to wear the little helmet with a camera on it,” says Lake. “But after she turned 2, it turned into a little bit of a debate. She’d say, ‘No, you wear it, Papa!’ ” As a psychologist—and a parent—Lake realized that an incentive might be in order. “I attached a horn to the helmet so when she puts it on, she becomes Luna-corn,” he says with a laugh. “Now the data is coming in much faster.”
—Jenny Comita, published under the title "Baby Steps" for Scope, NYU's Research Magazine, Issue 6, 2025
Related Articles
Global TIES Hosts ‘Early Childhood Matters’ Launch Event
Attendees from NYU, multilateral development banks, and early childhood NGOs and foundations discussed parental support initiatives.
Spanking and Other Physical Discipline Lead to Exclusively Negative Outcomes for Children in Low- and Middle-Income Countries
A study led by Assistant Professor of Applied Psychology Jorge Cuartas found physical punishment is linked to behavior and health problems, poor academic outcomes, and impaired social-emotional development.
Niobe Way's New Book on What Teenage Boys Teach Us
The professor of applied psychology explores what “boy” culture teaches us about the roots of mental illness, loneliness, and violence—and offers listening with curiosity as a solution.
Related Programs
Related Department
Applied Psychology
246 Greene Street , New York, NY 10003
P: 212-998-5555
E: applied.psychology@nyu.edu