Afterschool Programs That Follow Evidence-Based Practices to Promote Social and Emotional Development Are Effective

The purpose of this brief is to summarize the findings from our research review, which indicated that afterschool programs that follow four evidence-based practices are successful in promoting young people’s personal and social development (Durlak, Weissberg, & Pachan, 2010). While a number of afterschool programs need to change and improve, others have positively improved multiple dimensions of student learning and development. For this reason, the findings from various outcome studies on afterschool programs have led commentators to emphasize that a main focus in research should now primarily be to understand the factors that distinguish effective from ineffective programs in order to guide future policy and practice (Granger, 2010).


For example, the 21st Century Community Learning Centers initiative is an important large-scale funding stream for afterschool and summer learning in high-poverty schools and neighborhoods across America. Because the Community Learning Centers initiative allows for local design and variation, it should not be surprising that program results vary. Nor should it be surprising that early studies, conducted before the field was informed about promising and evidence-based practices and design, found mixed results. For instance, the large-scale evaluations of the outcomes of 21st Century Community Learning Centers programs serving elementary (James-Burdumy et al., 2005) or middle school students (Dynarski et al., 2004), that is centers that received federal funding through No Child Left Behind legislation, have generated controversy and led to questions regarding the wisdom of federal funding for afterschool programs. These early evaluations failed to detect any significant gains in achievement tests scores, although there were some gains in secondary outcomes such as parental involvement in school and student commitment to work. However, researchers have noted several methodological problems in these evaluations that involve the lack of initial group equivalence, high attrition among respondents, low levels of student attendance, and the possible nonrepresentativeness of evaluated programs (Kane, 2004; Mahoney & Zigler, 2006). 


There is also the critical issue of treating programs collectively as though they provided the same uniform set of services when this is clearly not the case. While some of these 21st Century Community Learning Centers provided students with intensive small-group instruction or individual tutoring, which has been shown to be an effective approach (Lauer et al., 2006), others provided relatively unstructured homework time, which is not likely to be successful. It is precisely because afterschool programs vary in form, structure, and specific goals that they should be carefully evaluated along these dimensions. There is no question that many young people and their families need and want expanded opportunities such as those funded by the 21st Century Community Learning Centers initiative. So the question should be not whether they should be offered, but rather what research-based design elements should be included to make them and other afterschool programs like them more successful. 


Our review included 68 studies in which those attending an afterschool program that had the specific goal of fostering personal and social development were compared to nonparticipating control youth. We did not review programs that focused exclusively on academic achievement. The reviewed programs were drawn from across the country; they operated in urban and rural areas and served school-aged youth between 5 and 18 years old. 


We hypothesized that effective programs would use evidence-based practices for enhancing young people’s personal and social skills. We were able to identify four practices used in some afterschool programs, but not in others. These four evidence-based practices formed the acronym SAFE and are explained further in our full research report. In brief, our procedures identified whether or not program staff used a step-by-step training approach (S), emphasized active forms of learning by having youth practice new skills (A), focused specific time and attention on skill development (F), and were explicit in defining the skills they were attempting to promote (E). Each of these practices has a strong research base in many skill training studies of youth. The afterschool programs that followed all four recommended practices were called SAFE programs (N = 41) and those that did not were called Other programs (N = 27). 


Our findings were clear-cut. SAFE programs were associated with significant improvements in self-perception, school bonding and positive social behaviors; significant reductions in conduct problems and drug use; and significant increases in achievement test scores, grades, and school attendance. The group of Other programs failed to yield significant improvements on any of these outcomes. Table 1 contains the mean effect sizes achieved on these outcomes by SAFE and Other programs. 


Table 1. Mean effect sizes on different outcomes for participants in SAFE and Other afterschool programs.


Note: All of the outcomes associated with SAFE programs but none of the outcomes for Other programs were statistically significant.


Figure 1. Average percentile gains on selected outcomes for participants in SAFE and Other afterschool programs.


Another way to portray how much of a difference in outcomes is associated with SAFE programs is by calculating an improvement index (Institute of Education Sciences, 2008). The improvement index is a percentile figure that suggests how much change the average youth would demonstrate depending on whether they participate in a SAFE or Other type of program. These percentiles are presented in Figure 1 for some notable outcomes from our review. For example, on average, youth could gain 8 percentiles in standardized test scores, show an increase of 11 percentiles in positive social behaviors (e.g., cooperation, helping others), and show a reduction of 12 percentiles in problem behaviors (e.g., aggression, noncompliance) if they were in a SAFE program. In contrast, participants in Other programs would show very minimal and statistically nonsignificant percentile improvements in each of these categories. 


Are such percentile gains worthwhile to participating youth? Of course, it would be preferable if the SAFE program outcomes were higher, but the outcomes for SAFE programs are comparable to those obtained by many other successful youth programs that have been carefully evaluated. For example, in terms of increasing positive social behaviors, reducing problem behaviors and promoting academic achievement, the outcomes for SAFE programs are similar to those achieved by many effective school-based programs designed to improve student academic performance or social adjustment (see Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011). In other words, afterschool programs that follow evidence-based skill training practices are part of the array of worthwhile interventions for youth. Our findings also suggest the possibility of aligning effective interventions during the school day with those occurring after school to maximize the benefits for participating youth.


The practical implications of our findings are that policy and funding should be focused on assisting more afterschool programs to develop evidence-based practices that are associated with better outcomes. As others have noted, quality matters in afterschool programs, just as it matters in other types of youth services (Hirsch, Mekinda, & Stawicki, 2010; Yohalem & Wilson-Ahlstrom, 2010). Carefully done evaluations can help us understand how quality is manifested in afterschool programs that vary in their structural and operational characteristics and in relation to different participant outcomes. With the knowledge that we now have, we should spend time and energy developing strategies, supports, policies, and funding to expand SAFE afterschool and summer learning programs through the 21st Century Community Learning Centers initiative and similar initiatives where they are needed across America rather than continue to argue whether they make a positive difference. 


References


Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405−433. 


Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294–309. 


Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., & Mansfield, W. (2004). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers Program: New findings (NCEE 2004-3001). U.S. Department of Education, National Center for Education Evaluation and Regional Assistance. Washington, DC: U.S. Government Printing Office.


Granger, R. C. (2010). Understanding and improving the effectiveness of after-school practice. American Journal of Community Psychology, 45, 441–446. 


Hirsch, B. J., Mekinda, M. A., & Stawicki, J. A. (2010). More than attendance: The importance of after-school program quality. American Journal of Community Psychology, 45, 447–452.


Institute of Education Sciences. (2008). What Works Clearinghouse procedures and standards handbook. Retrieved from http://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19 


James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., Mansfield, W., & Pistorino, C. (2005). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers Program: Final report (NCEE 2005-3002). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.


Kane, T. J. (2004). The impact of after-school programs: Interpreting the results of four recent evaluations. Retrieved from http://www.wtgrantfoundation.org/


Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Green, M. (2006). Out-of school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.


Mahoney, J. P., & Zigler, E. F. (2006). Translating science to policy under the No Child Left Behind Act of 2001: Lessons from the national evaluation of the 21st-Century Community Learning Centers. Journal of Applied Developmental Psychology, 27, 282–294.


Yohalem, N., & Wilson-Ahlstrom, A. (2010). Inside the black box: Assessing and improving quality in youth programs. American Journal of Community Psychology, 45, 350–357.