Sample of research projects currently underway by Q-TRAIN faculty mentors
This page describes the types of research projects in which fellows may have the opportunity to become involved.
Projects funded by IES that are currently underway
Practical solutions for missing data.
Missing data are endemic to virtually all empirical education research. However many researchers still rely on ad-hoc solutions to handle missing data (such as listwise deletion or combinations of mean imputation and inclusion of missing data indicators). These simple approaches typically throw away important information and can result in biased estimates. Multiple imputation (MI; Rubin, 1987) is an increasingly popular approach for missing data (Schafer, 1999; Raghunathan et al., 2001; van Buuren et al, 2006; van Buuren, 2007). However there are several outstanding issues with regard to use of MI, particularly for more applied researchers. MI often involves complex models whose fit can be difficult to evaluate; it’s difficult to know what’s going on “under the hood” (Abayomi, Gelman, and Levy, 2008; Su, Gelman, Hill, and Yajima, 2011; Stuart et al., 2009). MI is run using software that can be tedious and is not always user-friendly. General-purpose MI software has yet to be extended to more complex data structures such as panel data and hierarchical data structures with missing data at the group level. Finally MI packages typically make the restrictive assumption that data are missing at random. We are working on developing software for MI that provides the user with a more transparent, interactive interface and allows the researcher to better monitor, diagnose, and fix problems with imputations. Moreover we will be extending it to accommodate more complex data structures and missing data mechanisms. Of particular relevance to IES, we are also interested in exploring missing data solutions for specialized settings encountered frequently in education research. For instance we are planning to develop a specialized module for imputing data generated in the context of a randomized experiment under each of the following conditions: missing predictor data only, missing outcome data only, missing predictor and outcome data. We are also in the process of expanding the imputation models available to accommodate multilevel data structures under each of the following conditions: missing individual-level data only, missing group-level data only, missing individual-level and group-level data.
Practical tools for multilevel modeling.
Multilevel data structures are common in education research in studies that range from descriptive analyses of children nested within classrooms and schools to more formal cluster-randomized field experiments (Raudenbush and Bryk, 2002; Kreft and De Leeuw, 1998; Gelman, 2006). The range of hierarchical models that standard software fits readily, however, is quite narrow relative to the complexity of many real-life studies. Moreover, a problem frequently encountered when fitting such models using standard software is that the (restricted) maximum likelihood estimates are on the boundary of parameter space, or that convergence fails. In either case, these standard models and fitting procedures can yield estimates that do not make sense. For example, variance components may be estimated as zero, correlations between random intercepts and random slopes may be estimated as 1 or -1, and non-convergence may result due to sparseness or separation. We are currently 1) developing a class of statistical models for nested and non-nested structures in the context of ongoing education research projects, 2) to fit these models using a mix of classical and Bayesian approaches, 3) to implement the fitting procedure in the widely used statistical packages R and Stata, and 4) and to develop and implement tools for checking model fit, building confidence in the models, and interpreting each model in the context of the model building process. Doing this research involves several tasks and areas of expertise, including: statistical theory as it relates to the construction of classes of models, research in statistical computing; programming and interface development; and work on active educational research projects.
Tools for analyzing sensitivity of inferences to departures from standard assumptions.
Researchers often find themselves in the position of trying to draw inferences from imperfect data, whether from “broken” randomized experiments or observational studies in which randomization never occurred. We are developing methods to assess sensitivity to departures from the standard assumptions in several key types of scenarios that education researchers encounter. We want to assess sensitivity in the context of observational studies to deviations from the selection on observables assumption (building on work by Rosenbaum, 1987; Gastwirth et al., 1998; Rosenbaum, 2002; Imbens, 2003; Altonji et al., 2005), to failures of the assumption of common support (sufficient overlap in covariate distributions) across treatment groups (Hill, 2011b), and to departures from the parametric assumptions implicit in traditional analytic models (Hill, 2011a). In the context of (broken) randomized experiments that lack full compliance with treatment assignment, researchers often use instrumental variables techniques to estimate local average treatment effects for those who would comply with their assignment, however these methods themselves rely on untestable assumptions (such as the exclusion restriction). Therefore, building on previous work (Hirano et al. 2000) we will also develop strategies and software to help researchers assess the sensitivity of their estimates to deviations from these assumptions. In all areas we will develop user-friendly software including graphical displays of the findings that will provide deeper intuition regarding the results.
Education research being performed by colleagues with whom we can collaborate.
Fellows will have an opportunity to participate in education research underway at the two universities. The PI’s have a history of collaboration with faculty and students, at both Teachers College (Columbia University) and NYU Steinhardt, who conduct research at the intersection of education with economics, sociology or psychology and seek to utilize cutting-edge research designs and analyses. The PIs are currently working with a number of live data analysis problems in education research and are also currently developing software in R, Stata, and C++ for multiple imputation, multilevel modeling, and Bayesian computation, so there are places for postdocs at all levels to apply and improve their skills in research methods, data analysis, and computation. A theme throughout these projects is the development of technically sophisticated methods with the goal of fitting more realistic, complicated models for educational studies, and also developing the tools for evaluating and building confidence in these models. A secondary emphasis is on providing tools for less sophisticated users to make use of the more complicated models or algorithms that have been created in ways that minimize potential abuse of the methods and maximize their understanding of the assumptions involved. This research has the potential to be highly influential and an important part of the educational research infrastructure.
In working on these projects, fellows will be developing their research skills in ways that will enable them to both develop their skills in methodological development and become engaged in the substance of the educational problems being studied. Moreover, these experiences will prepare them to develop proposals and projects of their own, using state-of-the-art methods with live research problems.
Potential future projects
Hierarchical modeling for group randomized experiments.
Multilevel data structures are common in education research in studies that range from descriptive analyses of children nested within classrooms and schools to more formal cluster-randomized field experiments. The range of hierarchical models that standard software fits readily, however, is quite narrow relative to the complexity of many real-life studies. This causes problems when researchers choose their models for data analysis based on computational or data limitations rather than on the phenomena and decision problems of substantive relevance. We are in the process of developing and evaluating new families of models as well as new methods to estimate and validate such models reliably. In particular, penalized likelihood approaches allow us to avoid directly estimating the group-level variation as a way to improve efficiency and reliability of estimated treatment effects and comparisons.
Survey weighting and poststratification as tools for generalizing inferences to a larger population.
The theory of causal inference often focuses on estimation of the local average of sample-specific treatment effects, but in the presence of treatment interactions, it is often of more interest to estimate an average effect over a larger population, not merely those in the study. To do so, we are working on methods for estimating effects conditional on the interaction of many pre-treatment covariates and then poststratifying over the population, combining as necessary with other sources of survey information. These methods can help with generalizing the results from smaller studies to larger, and potentially more heterogeneous populations. Part of this work is focused on identifying subgroups to whom we should be cautious in extrapolating results. So far this research has used regression models with multilevel structure that allows stable adjustments for many poststratification factors. Further extensions of this work will involve fitting models based on Bayesian additive regression trees that relax the parametric assumptions of standard multilevel models.
Hierarchical modeling and multiple comparisons.
When estimating treatment effects on subgroups, making comparisons between several groups, or evaluating multiple outcome measurements, worries arise about the validity of significance tests in the presence of multiple comparisons. We are developing methods for addressing multiple comparisons using multilevel models that use partial pooling to get appropriate coverage. Compared to classical procedures, hierarchical models for multiple comparisons adapt to the data and typically allow for far greater power when testing for significant effects; this can be particularly important when attempting to identify subgroup effects.
Models for longitudinal categorical data with large state spaces.
Models for categorical data have a long history in statistics and social science (Goodman ,2007; Huang, 1998; Hu and Li, 2009), but modern social processes such as movement through postsecondary education classwork offer many new methodological challenges. Many of these challenges stem from the categorical nature of the observations, which are inherently hard to compare. We are exploring several related strands of research associated with educational “pathways” or progression (Scott and Kennedy, 2005; Zeidenberg and Scott, 2011). The first strand is identifying programs of study from partial transcript data. This allows one to form comparable groups so that at a minumum, evaluation of program successes can be made across populations. A second strand involves assessing the “coherence” of course progression for advisement or evaluation purposes. Classification and clustering techniques, including mixture modeling, have been explored, and from the modeling approaches new metrics applied to educational pathways will be evaluated for descriptive, advisement and evaluation purposes.