Improving the Quality and Impact of Afterschool and Summer Programs: Lessons Learned and Future Directions

The afterschool field has made important progress in the past 15 years, particularly since the federal 21st Century Community Learning Centers Program rapidly expanded starting in 1998. Increased federal, state, and local support demonstrates that taxpayers and policy makers want safe and engaging activities for young people while parents work. As a result, afterschool options have grown rapidly, with programs adding spaces and expanding to new sites. At the same time, funders and practitioners have created infrastructure—namely state and local intermediary organizations—to advocate for the field and support its expansion. 


The evidence that afterschool programs can deliver on multiple goals—academic, social, and behavioral—is much stronger than it was 15 years ago. However to produce positive effects, programs must be effectively designed and delivered. As afterschool and summer learning programs have made a greater claim on public resources—and the economy has tightened—they are experiencing increased pressure to justify support. The prevailing view seems to be that if these and other social programs are going to draw significant funding, they need to be able to produce positive results consistently. Similar accountability pressures have occurred in other sectors such as preschool, K–12 education, and mentoring. One result of this pressure is increased attention to program quality within the field; and as we discuss later, a great deal that has been learned in that regard is now being incorporated into afterschool and summer program design and delivery.


Some of the pressure on afterschool has come from K–12 education, which is itself being pushed to improve achievement and attainment. As interest grows within the education community about how afterschool and summer programs can play a role in “expanded learning” efforts, challenging questions related to mission persist for providers. What should be the focus of afterschool programs? Is developing “21st century skills” such as personal responsibility, teamwork, and persistence paramount, or should programs be concerned with a narrower set of academic outcomes? Should programs be operated by schools, community organizations, or both? Should services be delivered in school buildings or elsewhere in the community? 


Although research does not provide clear answers to these questions, in part due to variation in local needs, goals, and program design, it does affirm the increased focus within the field on defining and improving program quality. As noted above, programs can have positive effects on academic, social, and behavioral outcomes, but not all programs that set out to achieve such effects do so, and we know that quality varies both within and across sites. Understanding why this is so has become an important priority. Are varying results due to program content? Program processes and structure? Characteristics of the organization implementing the program (for example, how well that organization is run or its rates of staff turnover)? Features of the surrounding community (for example, youth being able to safely get to the program regularly)? 


While much more needs to be learned, especially about how organizational and community factors affect afterschool program effectiveness, current research does confirm a consensus among practitioners—that program processes, content, and structure matter. Focusing on these features has some important advantages. In contrast to community- or family-level factors, program-level features are under the control of practitioners, and thus afterschool supervisors and line staff consider them a “fair” focus for accountability. Significant progress has been made on identifying these features of program effectiveness, designing valid and reliable ways to measure them, and helping program leaders and staff assess and improve them. The remainder of this article reviews this progress and discusses our recommendations for advancing the afterschool field. 


Developments in the Field


An evolving evidence base. As noted above, evidence that afterschool programs can deliver on multiple goals—academic, social, and behavioral—is much stronger than it was 15 years ago. Reviews conducted by Patricia Lauer and colleagues (2006) and Joe Durlak and Roger Weissberg (2010) were particularly useful in synthesizing the results from a large number of program evaluations. Durlak and Weissberg’s review had a major influence on the field. They found, on average, programs had a positive effect on a range of academic and other important outcomes. They also began to shed light on why. They reviewed 68 evaluations of afterschool programs focused on improving personal and social skills, such as reducing risky behavior. The results drew attention to the importance of specific program features (for example, implementing active and sequential activities focused on explicit goals) in producing positive effects. Lauer and her colleagues reviewed 35 evaluations of academically focused afterschool and summer programs for low-income children. They, too, found positive news on academic measures, although they were not able to identify particular program or contextual features that predicted the positive effects beyond participation itself. 


Positive effects in both reviews were driven by a subset of the programs in the sample (roughly one-third), and the evaluations included in both reviews were of relatively small programs. Among the few large-scale programs that have been rigorously evaluated, their effects are limited (Granger, 2011), and we still lack a clear understanding of why this is so. There is increasing interest in this question, however, and efforts are underway in the field to address this.


Increased understanding of high quality practice and how to measure it. Research shows that interactions among young people and adults during program activities are positively related to how well youth function and their developmental outcomes. Practitioners tend to see this as more than a useful correlation and believe that staff-youth interactions are the active ingredients that distinguish programs that make a difference from those that do not (Yohalem & Wilson-Ahlstrom, 2009). Research has not yet proven this, but it is beginning to make the case that adult/youth interactions cause youth outcomes to change. Historically, though, monitoring and accountability have focused on structural features such as staff qualifications and staff-student ratios that do not seem to predict effectiveness, at least in the K–12 research literature (Mashburn et al., 2008). 


Durlak and Weissberg’s analyses illuminated the importance of specific program features that might productively shape staff-youth interactions, and they did so at a time when the afterschool field was ready to listen. In 2002, the National Research Council (Eccles & Gootman, 2002) identified eight features of positive developmental settings. Since then, consensus has been building about what constitutes high quality practice in afterschool settings and how to measure it. The NRC report, along with research by Reed Larson, Deborah Vandell, Durlak and Weissberg, and others contributed to this growing consensus. By 2006, the Youth Program Quality Assessment (Youth PQA), developed by Charles Smith and colleagues at the HighScope Educational Research Foundation, was one of several observational tools designed to measure program quality being refined and used in the field to advance both research and practice. 


Systemic efforts to improve quality. As measures of program quality matured, practitioners leading afterschool organizations and systems, who were eager to use research-based tools to improve their programs, began incorporating them into their staff development efforts. Increasingly, continuous quality improvement systems that include observational assessments, improvement planning, and targeted training and coaching are being implemented and enhanced at the local and state levels. 


In addition, there is now limited, but promising evidence that such strategies can improve afterschool program quality. This echoes recent positive results (Allen, Pianta, Gregory, Mikami, & Lunl, 2011) in K–12 education about the impact of coaching-based professional development built around a validated tool for assessing teacher-student interactions. A rigorous evaluation of the Youth Program Quality Intervention—designed to improve practices measured by the Youth PQA—resulted in improved instruction and higher levels of staff retention in a wide range of afterschool sites (Smith et al., 2012). Designed to be responsive to the specific conditions of the afterschool field (for example, high turnover, limited training, part-time staff), the Youth Program Quality Intervention is a “low-stakes” model. Site managers are accountable for implementing continuous improvement practices rather than attaining specific thresholds of performance. 


This model is now being adapted and used by more than 80 networks of afterschool and summer programs across the country, including nine state education agencies using it to support implementation of the 21st Century Community Learning Centers initiative in their states. In Michigan, for example, observational assessments of all 21st Century Community Learning Centers grantees are conducted annually using the Youth PQA, and corresponding professional development offerings are available to all grantees. A network of regional coaches provides training, coaching, and technical assistance to a subset of grantees that either refer themselves or are referred by the Michigan Department of Education (MDE). Coaches work with those sites to implement improvement plans and maintain online service logs that are accessible to MDE. Coaches are in regular communication with MDE so that quality and compliance issues can be identified and addressed quickly. Several states, including Arkansas and Vermont, have developed an integrated quality improvement system based on the Youth Program Quality Intervention that supports both 21st Century Community Learning Centers and TANF-funded school-age child care programs. 


Recommendations for Advancing the Field


In order to get more afterschool and summer programs consistently producing robust, positive effects for children and youth, efforts to advance research and practice should build on the progress we have described. 


• Research. Instruments designed to measure program quality could benefit from revisions to make them more clear and specific. Researchers also need to produce better, scalable measures of youth behaviors and dispositions that contribute to school success, such as work habits, persistence, and engagement in learning, and others that push beyond the academic domain. More studies that assess how quality improvement approaches affect program practices are also needed (in general, results about the effectiveness of staff development programs in K–12 are mixed [Yoon et al., 2007]). Additionally, studies that confirm the belief that when staff practices improve, youth outcomes also improve would constitute a critical milestone for the field. Less likely to advance our understanding of how to improve quality are more impact evaluations of small programs. We know such evaluations can be an important gateway to gaining funding support. We already know, however, that such programs can work, but many need to do better—some much better—and the field is now headed down a fruitful path of better understanding how to define, support, and sustain high quality. 

• Practice. Practitioners need more validated, cost-effective approaches for continuously improving practice. One promising approach is for practitioners to partner with researchers to develop and test different improvement approaches. Such partnerships allow for the integration of research-based tools and knowledge with local circumstances and expertise, and the current press for evidence-based practice across the human services fields could help sustain such collaborations.

Partnerships should pioneer and test different approaches, including new assessment strategies and intentional variations in the duration and delivery of coaching and training. Rapid but disciplined research and development processes that advance practice and accrue reliable information about how to improve program quality are needed. This will require developing and institutionalizing new ways of working collaboratively across practice and research; promising strategies are being refined in health care and increasingly tested in education and human services to do just that (Bryk, Gomez, & Grunow, 2010). State afterschool networks can work with state education agencies and their research and evaluation partners to test and refine promising strategies within 21st Century Community Learning Centers-funded and TANF-funded school-age child care programs. Municipal afterschool systems can do the same at the local level, and national intermediaries can spread the word about promising practices. 

• Policy and funding. Afterschool and summer learning programs and systems are not able to bear the full cost of this important work of improving quality on their own. Public systems should allocate professional development and monitoring resources toward continuous improvement approaches; and foundations that support programs, infrastructure, and research should seize the opportunity to subsidize the development of tools and strategies designed to support continuous improvement. 


The afterschool and summer learning field is ripe for a focused wave of research and development that does not involve dramatic changes but rather capitalizes on the significant progress made over the past 15 years. Afterschool and summer learning programs can have positive effects on a range of important outcomes, and thus they have earned the right to be included in discussions about advancing young people’s learning and development. Future investments in education and youth development should recognize afterschool and summer as important opportunities to advance student success, and more fully capitalize on growing capacity at the state and local levels to expand and improve programs. 


References


Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun J. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333, 1034–1037.


Bryk A. S., Gomez L. M., & Grunow A. (2010). Getting ideas into action: Building networked improvement communities in education. Stanford, CA: Carnegie Foundation for the Advancement of Teaching. 


Durlak, J., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294–309. 


Eccles, J., & Gootman, J. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press.


Granger, R. (2011). Our work on the quality of after-school programs: 2003–2011. William T. Grant Foundation Annual Report. New York, NY: William T. Grant Foundation. 


Larson, R. W., Rickman, A. N., Gibbons, C. M., & Walker, K. C. (2009). Practitioner expertise: Creating quality within the daily tumble of events in youth settings. In N. Yohalem, R. Granger, & K. Pittman (Eds.), New Directions for Youth Development (No. 121, pp.71–88). San Francisco, CA: Jossey-Bass. 


Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.


Mashburn, A. J., Pianta, R. C., Hamre, B. K., Downer, J. T., Barbarin, O. A., Bryant, D., . . . Howes, C. (2008). Measures of classroom quality in prekindergarten and children’s development of academic, language, and social skills. Child Development, 79, 732–749.


Smith, C., Akiva, T., Sugar, S. A., Lo, Y. J., Frank, K. A., Peck, S. C., . . . Devaney, T. (2012). Continuous quality improvement in after-school settings: Impact findings from the Youth Program Quality Intervention study. Washington, DC: Forum for Youth Investment.


Vandell, D. L., Shernoff, D. J., Pierce, K. M., Bolt, D. M., Dadisman, K., & Brown, B. B. (2005).Activities, engagement, and emotion in after-school programs (and elsewhere). New Directions for Youth Development, 105, 121–129.


Yohalem, N. & Wilson-Ahlstrom, A. (2009). Measuring youth program quality: A guide to assessment tools (2nd ed.). Washington, DC: Forum for Youth Investment. 


Yoon, K., Duncan, T., Lee, S., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how teacher professional development affects student achievement. Washington, DC: Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest.