A PRIISM Seminar by MIT's Naoki Egami
Abstract
Social scientists use automated annotation methods, such as supervised machine learning and, more recently, large language models (LLMs), that can predict labels and generate text-based variables. While such predicted text-based variables are often analyzed as if they were observed without errors, we show that ignoring prediction errors in the automated annotation step leads to substantial bias and invalid confidence intervals in downstream analyses, even if the accuracy of the automated annotations is high, e.g., above 90%. In this talk, we introduce two complementary strategies to conduct valid statistical inference with AI-generated variables. First, we introduce a framework of design-based supervised learning (DSL) that can provide valid statistical estimates, even when predicted variables contain non-random prediction errors. DSL employs a doubly robust procedure to combine predicted labels and a smaller number of expert annotations. DSL allows scholars to apply advances in LLMs to social science research while maintaining statistical validity. Second, we propose debiased inference with multiple imperfect measurements (DMM), a framework that combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels. Building on the established results on CP decomposition, DMM assumes that these measurements are independent conditional on the latent true label and observed unit-level features, such as text features represented by embeddings. This framework allows for unknown misclassification rates to vary across annotation methods (e.g., large language models) and across units of annotation (e.g., texts). We use various social science applications to illustrate their general applicability and when to use which method.
Papers
- Using Large Language Model Annotations for the Social Sciences: A General Framework of Using Predicted Variables in Downstream Analyses, 2024
- Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements, 2026
Bio
Naoki Egami is an associate professor in the Department of Political Science at the Massachusetts Institute of Technology. He is also a faculty affiliate of the Institute for Data, Systems, and Society (IDSS) at MIT. He specializes in political methodology and develops statistical methods for questions in political science and the social sciences. Specifically, he works on causal inference and machine learning methods.
