Center for Practice and Research at the Intersection of Information, Society, and Methodology (PRIISM)

Event

Validating language model simulations for behavioral science

Register to Attend In-Person Register to Attend Online
Join PRIISM and Dr. Jessica Hullman to explore the promise and pitfalls of using AI and large language models as surrogates for human research participants.
Jessica Hullman

A PRIISM Seminar by Northwestern University's Jessica Hullman

Abstract

A growing literature positions AI, and especially large language models (LLMs), as a transformative technology for simulating human behavior in surveys and experiments. This promise sparks a debate over how best to leverage this new data source. While some empirical studies report that LLM-simulated responses can approximate patterns found in real survey and experimental data, others present mixed results. One source of confusion is a lack of consensus on what it means to show that AI surrogates are valid for the purpose of studying human behavior. Dr. Hullman will contrast heuristic approaches to validation in the literature from statistical calibration approaches that account for biases learned on items for which human ground truth labels are known, and discuss remaining challenges to using these approaches in practice. She will also reflect on how meta-scientific questions arise about the epistemic status of silicon samples as behavioral data.

Bio

Jessica Hullman is Ginni Rometty Professor of Computer Science and a Fellow at the Institute of Policy Research at Northwestern University. Her research develops methods for evaluating and improving AI and ML systems for human moderated decision-making and scientific inference. Jessica’s work has been awarded multiple best paper awards at top conferences, a Microsoft Faculty award and NSF CAREER, Medium, and Small awards as PI, among others.

NYU provides reasonable accommodations to people with disabilities. Please submit your request for accommodations for events and services at least two weeks before the date of your accommodation need. Although we can't guarantee accommodation requests received less than two weeks before the event, you should still contact us and we will do our best to meet your accommodation need. Please email susana.toro@nyu.edu for assistance.