Design2Learn: Implementation and Impact Study

Cheri Fancsali (November 2021)

In New York City and around the country, there are high hopes for using the afterschool hours to enhance students’ academic engagement and performance. Design2Learn (D2L) is a NYC-based afterschool program aimed at increasing middle school students’ interest, engagement, and achievement in science. Designed by ExpandED Schools, D2L centers around three core strategies: 1) collaborative teaching between an in-school science teacher and two afterschool educators, 2) curricular bridging, which connects students’ in-school science instruction with the afterschool activities, and 3) design-based learning, which emphasizes hands-on projects and inquiry-based instruction. To support the implementation of D2L, ExpandED Schools and its partner, the New York Hall of Science (NYSCI), provided educators with intensive professional development, including week-long summer institutes, Saturday workshops, strategic planning sessions, and on-site coaching.

In 2016, the U.S. Department of Education Investing in Innovation (i3) program awarded ExpandED Schools a grant to support the design and evaluation of D2L, with the Research Alliance serving as the external evaluator. Our study randomly assigned 32 afterschool programs to offer D2L or be part of a control group that offered their usual science programming. We examined D2L’s implementation, as well as its impact on various student outcomes. Among our key findings:

• Overall, we found strong fidelity of implementation, with sites participating fully in the PD and support that was provided. D2L sites implemented the three key pillars of the model—collaborative teaching, curricular bridging, and design-based instruction—and overall program quality was rated highly.
• Differences between D2L and the control site activities were small. Control sites also offered students engaging science activities that were connected to their in-school curriculum, and program quality was rated similarly.
• Both D2L and control sites offered fewer hours of afterschool activities than expected, and student attendance was low. On average, D2L students attended about 32 hours of programming, and control students attended about 34 hours of programming—much less than the 72 hours per year called for by the D2L model. In addition, most students attended for only one year, rather than the three years originally envisioned by ExpandEd Schools.
• D2L had no effect on students’ interest and engagement in science, perseverance, or critical thinking behaviors, as measured by student surveys. It also had no effect on students’ science grades or school attendance.

Together, these findings suggest that there may not have been a substantial difference between the D2L and control experiences, and students may not have received adequate exposure to the intervention, to influence student outcomes. The findings raise important questions about afterschool interventions designed to increase middle school student interest and achievement in science. The difficulty getting middle school students to participate in afterschool programming, at both the D2L and control sites, echoes findings from past research. Could a stronger focus on recruitment and retention have increased student participation? Or is it unrealistic to think that middle school youth will attend an academically oriented afterschool program at the levels originally envisioned by D2L? It is possible that an intervention like D2L might be better suited for elementary students, whose afterschool attendance is often more reliable because of parents’ needs for childcare. Looking ahead, it will be essential to continue learning about what works to recruit and engage middle school youth in academically focused afterschool programs—and what type of programming can improve their outcomes, in science and in other subjects.