This evaluation was designed to examine the facilitators and barriers of EFA implementation. As such, we relied heavily on qualitative methods. A team of four researchers worked with BPS to identify four case study schools (Curley, King, Mendell, and Orchard Gardens) and conduct individual and group interviews with stakeholders involved in EFA. These schools represented a range of diverse factors, including the extent of implementation, type of implementation, varying school demographics, academic achievement, and neighborhood (see tables 6-21 for more information). Case study research was conducted during SY 2018-19, mainly in fall 2018. Additional data were collected during spring of 2019, including seven EFA principals, five EFA coaches, and three EFA support/ administrative staff.
Data were analyzed in two stages: preliminary analysis and focused analysis. The preliminary data analysis consisted of composing field notes and memos during and after each of the one-on- one interviews and focus groups. The focused data analysis occurred primarily after all the interviews were completed. All transcribed files were uploaded to Dedoose, a software program designed for computer-assisted qualitative and mixed-methods data, text, and multimedia analysis. The research team coded and themed the transcriptions from the interviews. Using Dedoose, the team conducted a focused analysis of the interview transcripts and analytic memos, identifying connections between the stakeholders across the data sample. Through this process, the researchers organized the data into the sub-themes that formed the main findings of the study.
It is necessary to caution how practitioners and researchers should use or apply the findings of this research. The qualitative findings presented in this section can be used to understand some highlights and challenges in the implementation and fidelity of EFA programming across school sites. However, the findings presented in this section are limited to a small sample of school teachers, principals, coaches, and parents. As such, these findings are not generalizable, as all members of the school communities were not represented equally.
Although this evaluation was primarily qualitative, we triangulated some of our qualitative data, using secondary data provided by BPS. We examined descriptive school-level changes in MCAS ELA scores from 2017 to 2019 among Black and Latinx 4th and 5th grade students. We chose to focus on Black and Latinx students, because EFA sought to offer greater opportunity to students from historically marginalized backgrounds. We chose to focus on 4th and 5th grade because EFA was implemented in 4th grade for three consecutive school years (SY 2016-17, 2017-18, and 2018-19) and in 5th grade for two consecutive school years (2017-18, and 2018-19). EFA did not begin in 6th grade until SY 2018-19, so we did not include 6th grade in our analysis, thus excluding the three middle schools from the analysis. Finally, we chose to focus on ELA scores and not math because only two schools implemented interventions directly related to math instruction, while eight schools implemented WIN reading blocks and nine schools implemented SFL writing, albeit to varying extents. Note that these analyses
are merely descriptive and do not control for variation in school-level qualities, such as demographics, free/reduced price lunch, size, etc. Because EFA implementation and fidelity varied so much across schools, much of our descriptive analysis focused on the relationship between EFA implementation and change in ELA scores.
Additionally, we investigated teacher satisfaction with the EFA initiative and its various components via survey in the Spring of 2019. Of the 143 EFA teachers across 16 schools, 63 completed the survey (a response rate of 44%). For this survey, teachers were asked to report their satisfaction levels with EFA and which aspects of the initiative they viewed as being successful. We then created summary descriptive statistics from teacher responses to the survey questions (e.g., mean response on a Likert scale item ranging from 0-5, with 0 being ‘Not at all satisfied’ and 5 being ‘Very satisfied’).
