Skip to main content
Join PRIISM and Dr. Naoki Egami to learn how to apply two complementary strategies to conduct reliable statistical inference with AI-generated variables in social science research.
Naoki Egami

A PRIISM Seminar by MIT's Naoki Egami

Abstract 

Social scientists use automated annotation methods, such as supervised machine learning and, more recently, large language models (LLMs), that can predict labels and generate text-based variables. While such predicted text-based variables are often analyzed as if they were observed without errors, we show that ignoring prediction errors in the automated annotation step leads to substantial bias and invalid confidence intervals in downstream analyses, even if the accuracy of the automated annotations is high, e.g., above 90%. In this talk, we introduce two complementary strategies to conduct valid statistical inference with AI-generated variables. First, we introduce a framework of design-based supervised learning (DSL) that can provide valid statistical estimates, even when predicted variables contain non-random prediction errors. DSL employs a doubly robust procedure to combine predicted labels and a smaller number of expert annotations. DSL allows scholars to apply advances in LLMs to social science research while maintaining statistical validity. Second, we propose debiased inference with multiple imperfect measurements (DMM), a framework that combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels. Building on the established results on CP decomposition, DMM assumes that these measurements are independent conditional on the latent true label and observed unit-level features, such as text features represented by embeddings. This framework allows for unknown misclassification rates to vary across annotation methods (e.g., large language models) and across units of annotation (e.g., texts). We use various social science applications to illustrate their general applicability and when to use which method. 

Papers

Bio

Naoki Egami is an associate professor in the Department of Political Science at the Massachusetts Institute of Technology. He is also a faculty affiliate of the Institute for Data, Systems, and Society (IDSS) at MIT. He specializes in political methodology and develops statistical methods for questions in political science and the social sciences. Specifically, he works on causal inference and machine learning methods.

NYU provides reasonable accommodations to people with disabilities. Please submit your request for accommodations for events and services at least two weeks before the date of your accommodation need. Although we can't guarantee accommodation requests received less than two weeks before the event, you should still contact us and we will do our best to meet your accommodation need. Please email susana.toro@nyu.edu for assistance.