Fifteen Years of Evaluation of 21st Century Community Learning Centers: A Driver for Program Quality and Capacity in the Field

The agreement by key congressional and administration leaders to significantly increase funding of the landmark federal 21st Century Community Learning Centers legislation between 1997 and 2001 was a powerful signal that afterschool programs and activities were worth significant public investment as part of the nation’s efforts to educate and prepare its children for future success. At the same time, the legislation’s evaluation requirements and the subsequent emphasis on “scientifically based research” in the 2001 No Child Left Behind Act (NCLB) made it clear that these newly funded programs had to be accountable and prove their public value.


These challenges, including their accompanying performance management and accountability requirements, were powerful drivers for taking data and evaluation seriously in a new field. Addressing these challenges was also a shared priority of the innovative public and private partnership begun in 1998 between the United States Department of Education and the C. S. Mott Foundation. The Foundation’s leadership, along with the significant national opportunity that the 21st Century Community Learning Centers initiative afforded for continuing support for afterschool and expanded learning opportunities for children and youth, leveraged subsequent philanthropic investment in evaluation. Without these strategic foundation investments, the afterschool field would not be in the strong position it is in today.


So what has all of this investment in evaluation helped the field achieve in the past 15 years? In 1997 there existed little by way of evaluation of afterschool programs. Since then, the federal investment in the 21st Century Community Learning Centers initiative, along with strategic evaluation investments by others, has built afterschool into a maturing field with demonstrated public value on an array of commonly valued youth outcomes. In this commentary, I offer a brief scan of the state of afterschool evaluation to suggest that the field is, in fact, maturing and has met the evaluation challenge. Quality afterschool programs that are well designed can positively impact areas on which they focus. I also suggest that the field’s evolving research and evaluation agenda holds important lessons for other fields.


For me, a mature field in the 21st century positions evaluation and performance management not only to show it delivers valuable public outcomes for youth but also to ensure it can continue to attain and be accountable for these outcomes. With respect to the position and role of evaluation, a maturing field has three distinct features: practitioners with a commitment to using information to support continuous improvement, innovation, and accountability; a substantial, high quality, and nuanced research and evaluation base from which to learn and to show the public the value of high quality programs; and a deepening research- and practice-based understanding of how to build the quality programs and activities that continue to deliver their promised outcomes.


The Harvard Family Research Project has been tracking and synthesizing the results of afterschool evaluations for over a decade. We developed and maintain a national database of afterschool program evaluations for the field (www.hfrp.org/out-of-school-time/ost-database-bibliography). Both the number and quality of the studies in the database and our understanding of the evolution of afterschool evaluation underscore how important the 21st Century Community Learning Centers initiative has and continues to be, not only in funding programs but also in creating and shaping the knowledge base for the afterschool field that can be used by school, community, and afterschool leaders, as well as public and nonprofit funders.


The evaluation of the 21st Century Community Learning Centers programs got off to a rocky start with a federally funded and premature outcome evaluation reporting mixed results in 2003. It was used by some at the federal level to attempt to reduce funding for the program by half; but fortunately, as other evidence was documented and the serious concerns about how this early evaluation was conducted became known, support in Congress and the administration was retained. By being conducted early on in the field’s development, despite the study’s flaws and because of the reaction of researchers suggesting problems with the study, the process actually helped clarify the role of evaluation and position it to be useful in developing this growing field, hence my assertion that it was premature. In particular, it suggested some programs were effective while others were not, thereby putting a critical and early emphasis not only on assessing outcomes but on understanding program goals and implementation and on determining the factors and conditions necessary to deliver quality and effective services (Evaluation Exchange, 2002).


The 21st Century Community Learning Centers initiative has created incentives for evaluating afterschool programs and has therefore shaped afterschool evaluation in a number of ways. It has funded and stimulated programs to conduct evaluation, reflected in the fact that at least a third of the programs in our database of afterschool programs call themselves 21st Century Community Learning Centers or indicate they receive some of their funding from this source. Because the 21st Century Community Learning Centers funding does not support any one model or approach to afterschool programs and activities, the initiative has stimulated the evaluation of a wide array of program models and approaches operating in diverse communities and conditions.


This decision not to fund a particular approach turns out to have been a wise one, not least because studies show that participation and engagement in afterschool depend on children and youth having choices among programs and access to diverse activities. The large number of 21st Century Community Learning Centers programs and their diversity have also attracted applied developmental researchers using afterschool programs as sites for studying where youth learn and what engages them in learning, thereby enriching the knowledge base of the field (Mahoney, Lord, & Carryl, 2005; Mahoney, Larson, & Eccles, 2005; Durlak, Mahoney, Bohnert, & Parente, 2010).


Multiyear funding support from the 21st Century Community Learning Centers initiative allows local program sites to test new and creative approaches and incorporate successful ones into their programming (see HFRP 21st Century Community Learning Centers Research Updates, 2010, 2012). Multiyear program support also allows flagship leaders in the afterschool field, such as the large, multiprogram, citywide organizations that serve large numbers of children and youth (for example, TASC in New York City and LA’s BEST), to attract evaluation support and develop a longer-term evaluation strategy. Their ongoing series of evaluations and partnerships with evaluators are important for the field because they address key questions about the professional training, organizational supports, and other elements of infrastructure and program quality that lead to positive outcomes (HFRP 21st Century Community Learning Centers Bibliography, 2010; Reisner et al., 2007; Huang et al., 2007).


At this point, with federal 21st Century Community Learning Centers and philanthropic support, the afterschool field has a large number of evaluations meeting the criteria NCLB set in 2001 for scientifically based research in education. There are many small, single-site evaluations, as well as large, multi-site evaluations, conducted by a growing national cadre of investigators who are using both experimental and quasi-experimental research designs to assess program outcomes. Having this large set of studies enables meta-analytic syntheses that examine outcomes across an array of programs and that tease out the success factors that enable positive ones (Durlak, Weissberg, & Pachan, 2010). There is also growing convergence across multiple studies on the success factors and elements of quality programs (Little, Weimer, & Weiss, 2008). The afterschool field is in a strong position because it can make evidence-based claims about its public value on an array of commonly valued youth outcomes, such as improved attendance, grades, homework completion, classroom participation, behavior and— depending on the focus—achievement and performance. These programs also contribute to an array of positive developmental outcomes, including socio-emotional skills and healthy behaviors that support learning, and they prevent a number of problem behaviors that are detrimental to school and life success (Little, Wimer, & Weiss, 2008).


Equally important, the afterschool field is benefitting from a steady flow of increasingly nuanced evaluations that have been providing information to address seven key questions that are critically important if it is to continue to grow and provide high quality services. I offer the questions here to invite others into a conversation about what the learning agenda for the field should contain and prioritize:


1. What works for whom, when, where, and why?

2. What doesn’t work?

3. What are the elements of high quality programs and activities?

4. How do the elements work together to achieve the desired youth outcomes?

5. What internal program organizational and leadership characteristics and processes are necessary to develop and maintain quality services?

6. What policy, funding, and infrastructure supports are necessary for high quality at scale?

7. How can and do afterschool programs fit together with schools, digital media, and other learning supports to offer coordinated, accessible, and seamless opportunities?


Many of the studies addressing the first three questions and some addressing number 4 are available in our searchable database and have been included in meta-analyses and key syntheses of the state of knowledge in the afterschool field (Lauer et al., 2006; Little, Weimer, & Weiss, 2008; Granger, 2010; Durlak, Weissberg, & Pachan, 2010; Durlak, Mahoney, Bohnert, & Parente, 2010). There are fewer research studies and evaluations to address questions 5 through 7. I suggest they are a priority for further research investment in the field and that addressing them will require the kinds of ethnographic and mixed methods work in the following examples.


Hirsch, Deutsch, and DuBois’s recent work (2011) exemplifies an important effort to understand the organizational dimension of service quality—an effort that is also being repeated in research across other education and human services domains (Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010; Glisson, 2007; Duggan, 2012; Douglass, 2011). Hirsch (2011) and his colleagues’ ethnographic work on three comprehensive afterschool centers examines how multiple organizational characteristics and processes like leadership, a strong focus on positive youth development, organizational climate, staff development and supervision, connections to family and community, and organizational learning all fit together and interact to create quality services that, in turn, move the needle on youth outcomes in a significantly positive direction. Their work is pathbreaking for the field in that it assembles the pieces that other studies have shown are important for quality services and shows how they all work together to create quality youth experiences.


The landscape of learning is rapidly changing, with more use of digital media and a growing emphasis on anywhere, anytime learning, both in and out of school. In this regard, another important new strand of work is being conducted by developmental researchers and ethnographers studying where and how youth use and learn with digital media. Both Baron’s (2006) work on self-initiated learning and Ito and colleagues’ (2009) studies of how youth use digital media, for example, highlight how youth are seeking opportunities to build important skills across learning environments, as well as how learning in school can lead to learning in afterschool and vice versa. It suggests that youth are actually ahead of institutions in seeking and connecting learning opportunities in and out of school and that they could both help make and benefit from greater connections.


Strategic investments in evaluation research over the past 15 years have yielded significant evidence that 21st Century Community Learning Centers and high quality programs that serve children and youth during the nonschool hours are essential for preparing young people for the future. 


It also shows what is essential to deliver high quality services that contribute to better learning and developmental outcomes for youth. In 15 years, the afterschool field has built a substantial research and evaluation literature that is serving as a driver for more high quality programs and opportunities around the country. It is also a model for how to invest in research and evaluation for those seeking to invest in building the knowledge base in other new service fields. That said–and as important as the knowledge we already have today is–we have work to do as a field to investigate and uncover findings about more complex aspects of this field from an organizational and systems perspective. The next frontier, in fact, includes more sophisticated research that studies expanded learning opportunities, including the perspective of children and youth themselves, and that reveals optimal ways to support learning processes, program capacity and scalability, and systemic infrastructure building. As this commentary suggests, the afterschool field is “on it.”


References


Baron, B. (2006). Interest and self-sustained learning as catalysts of development: A learning ecology perspective. Human Development, 49, 193–224.


Bryk, A. S., Sebring, P. B., Allensworth, E., Luppescu, S., & Easton, J. Q. (2010). Organizing schools for improvement: Lessons from Chicago school reform. Chicago, IL: University of Chicago Press.


Douglass, A. (2011). Improving family engagement: The organizational context and its influence on partnering with parents on formal child care settings. Early Childhood Research and Practice, 13(2), 1–20.


Duggan, A. (2012, March). Service is everything: How home visit service delivery impacts family outcomes. Harvard Family Research Project FINE Newsletter, 4(1).


Durlak, J. A., Mahoney, J. L., Bohnert, A. M., & Parente, M. E. (2010). Developing and improving after-school programs to enhance youth’s personal growth and adjustment [Special issue American Journal of Community Psychology, 45, 285–293.


Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal growth and social skills in children and adolescents. American Journal of Community Psychology, 45, 294–309.


Glisson, C. (2007). Assessing and changing culture and climate for effective services. Research on Social Work Practice. 17, 736–747.


Granger, R. C. (2010). Understanding and improving the effectiveness of after-school practice. American Journal of Community Psychology, 45, 441–446.


Harvard Family Research Project. (2002). Special report on the 21st Century Community Learning Centers program national evaluation. Evaluation Exchange, 9(1), 11–15.


Harvard Family Research Project, (2010, November). 21st CCLC bibliography: Evaluations and research studies of out-of-school time programs that receive 21st CCLC funding: Appendix to Research Update No. 4. Retrieved from http://www.hfrp.org/var/hfrp/storage/fckeditor/File/research4appendix.pdf

Harvard Family Research Project. (2010, November). Research update 4: 21st CCLC-funded afterschool programs. Retrieved from http://www.hfrp.org/publications-resources/browse-our-publications/resea... 21st-cclc-funded-afterschool-programs


Harvard Family Research Project (2012, May). Research update 8: 21st CCLCs—Stable funding for innovation and continuous improvement. Retrieved from http://www.hfrp.org/publications-resources/publications-series/research-... 21st-century-community-learning-centers-stable-funding-for-innovation-and-continuous-improvement


Hirsch, B. J., Deutsch, N. L., & DuBois, D. L. (2011). After-school centers and youth development: Case studies of success and failure. New York, NY: Cambridge University Press.


Huang, D., Coordt, A., La Torre, D., Leon, S., Miyoshi, J., Perez, P., & Peterson, C. (2007). The afterschool hours: Examining the relationship between afterschool staff-based social capital and student engagement in LA’s BEST (CSE Tech. Rep. No. 712). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).


Ito, M., Baumer, S., Bittanti, M., Boyd, D., Cody, R., Herr-Stephenson, B., . . . Tripp, L.. (2009). Hanging out, messing around, and geeking out: Kids living and learning with new media. Cambridge, MA: MIT Press.


Lauer. P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.


Little, P .M. D., Wimer, C., & Weiss, H. B. (2008). After school programs in the 21st century: Their potential and what it takes to achieve it. Issues and Opportunities in Out-of-School Time Evaluation, 10.


Mahoney, J. L., Larson, R. W., & Eccles, J. W. (Eds.). (2005). Organized activities as developmental contexts: Extracurricular activities, after-school and community programs. Mahwah, NJ: Lawrence Erlbaum.


Mahoney, J. L., Lord, H., & Carryl, E. (2005). Afterschool program participation and the development of childhood obesity and peer acceptance. Applied Developmental Science, 9(4), 202–215.


Reisner, E. R., Vandell, D. L., Pechman, E. M., Pierce, K. M., Brown, B. B., & Bolt, D. (2007). Charting the benefits of high-quality after-school experiences: Evidence from new research on improving after-school opportunities for disadvantaged youth. Washington, DC: Policy Studies Associates.