Often in education, the policy directions of school districts, state education systems, and the federal Department of Education call for and rely on the effective use of data. Such terms as “data-driven classrooms,” or “data-driven instruction” are common, and rightfully so. While the language comes from the business world, educators and policymakers alike reasonably ought to turn to the most robust, vetted, and current information available in decision-making from classrooms to campuses to the highest offices in our government. Decisions rooted in research are, ostensibly, more free of bias than decisions made on a whim or based purely on belief.
However, there lies a great danger in misrepresenting the significance of data when making broad claims about the direction of education policy and civil rights. Evidence of such misuse of data can be seen in this recent blog on the federal rollback of Office for Civil Rights (OCR) investigations under the direction of Secretary of Education Betsy DeVos appointee and acting head of the OCR Candice Jackson. According to Ms. Jackson, the rationale for the rollback of civil rights enforcement includes federal overreach, efficiency and clearing the backlogs.
The author of the blog, an established and respected conservative education reform advocate, makes the claim that because a recent study using data from all public schools in Arkansas found that poverty rate was more strongly associated with differences in exclusionary discipline practices, the scaling back of OCR investigations at the federal level is justified. His credentials notwithstanding, this is a prime example of some key ways data is misused to sway policymakers and the public on matters of educational policy. One achieves this by seizing onto information that in isolation appears to confirm a point of view as general proven fact rooted in data, a phenomenon known to quantitative researchers as “confirmation bias.” The problems with confirmation bias in arguments are multitudinous, not the least of which is that the evidence used is incomplete and insufficient to demonstrate the point.
One specific problem with the exclusive use of Arkansas data is that Arkansas, particularly Little Rock, is still fairly segregated. The fact is that, in 2014, more than 40% of the state’s Black population but less than 15% of its white population attended high-poverty schools. Therefore, the correlations identified in the study may have overlooked systematic differences in the types of administrators and policies favored when hiring administrators and staff for schools with heavy minority populations, which can entirely be influenced by racial bias at the district level, a fact which can be masked behind the variable “poverty,” labels like “tough school,” and the outcome “between-school differences” if one does not investigate how closely poverty is tied to race in schools. This would be the data error known as “omitted variables bias.” This is a particularly troublesome problem, because it can lead data to appear to over-explain- or conversely diminish the impact of- particular variables, due to incomplete identification of context, a constant challenge in quantitative research and analysis.
The final problem with the use of Arkansas data is that the correlations identified within Arkansas may not be representative of other states, much less the nation as a whole. Much of the research on school discipline and implicit bias shows that that the influence of race on disciplinary decision making in schools is undeniable. Therefore, if this is a national and systematic problem, looking to Arkansas alone to disprove the national trend is not enough.
If the problem is resources at OCR, the solution is reallocation of resources, not scaling back of a civil rights service. We cannot look to small, skewed, or non-representative data to justify changes to the enforcement of civil rights. Doing so makes trivial the struggle to bring these issues to the forefront and places too much faith in systems with histories of oppression and failure when it comes to underserved, underprivileged, and under-resourced populations. Assuming highly segregated schools will, if left unchecked, serve everyone equally, is indeed the opposite of “common sense.”
Evan M. Johnston is a doctoral student at NYU Steinhardt’s Teaching and Learning Department and a Graduate Research Assistant in the Center for Research and Evaluation at Metro Center. Follow him @evanmjohnston on Twitter.