PRIISM co-directors Drs. Marc Scott and Jennifer Hill along with visiting assistant professor George Perrett were awarded a $900,000 grant by the Institute of Education Sciences (IES). The grant, titled What when and for whom? Principled estimation of effect heterogeneity across multiple treatments, outcomes, and groups, aims to create a principled but flexible methodological framework for simultaneous estimation of treatment effects in the context of multiple treatments, outcomes, or moderators that mitigates threats to replicability and embed it in a user-friendly software interface.
Project Summary
Purpose: Causal inference is critical for education research because it informs decisions being made everyday by students, teachers, administrators, and policy makers. Methodological advances for estimating causal effects have grown considerably in the past few decades, however, when several treatments, outcomes, or moderators are involved, each analysis is still generally considered in a standalone way. This piecemeal approach to causal inference generally leads to overall underestimation of our true uncertainty about effect estimates and undermines our ability to detect replicable effects. We will create a principled but flexible methodological framework for simultaneous estimation of treatment effects in the context
of multiple treatments, outcomes, or moderators that mitigates threats to replicability and embed it in a user-friendly software interface.
Products: We will create an easy-to use software tool that mitigates many of the threats to replicability induced by multiple testing and researcher degrees of freedom in observational causal inference settings with multiple treatments, outcomes, or moderators. Our approach will simultaneously estimate multiple effects in randomized or observational studies with these complex elements in a way that not only diagnoses and reduces these threats, but appropriately accounts for uncertainty. This software will be embedded in a user-friendly tool that also includes a “wizard” to create pre-registration plans that make efficient use of the available data while controlling for researcher degrees of freedom.
Structured Abstract
Research Design and Methods: We will capitalize on the strengths of the existing Bayesian Additive Regression Trees (BART) framework for causal inference (Hill 2011; Hill, Weiss, and Zhai 2011; Carnegie, Dorie, and Hill 2019; Dorie, Perrett, Hill, and Goodrich 2022) and extend it to include the following features: multivariate models for multiple outcomes with explicit covariance structures, targeted priors for multiple treatments and subgroups, and principled strategies for detection of effect moderation. The BART framework will naturally allow for estimation of effects using flexible nonparametric response surfaces and current overlap diagnostics will be extended to handle the additional estimands induced by this framework. This software will be embedded in the existing thinkCausal framework – a causal inference tool that scaffolds analysis and provides opportunities for users to learn about the underlying concepts. A novel pre-registration wizard will help researchers develop principled but flexible analysis strategies, keeping them honest while allowing maximal use of existing data.
User Testing: Statistical tools are beneficial to the extent that they are used correctly. Fortunately our team has several years of experience building user-friendly tools and testing them to ensure that they work as intended. We will use strategies similar to those of our most recent IES grant, including: 1) gathering feedback from users on ease of use and 2) a randomized trial to assess whether researcher conclusions using the new tool are better than those reached with standard software.