Afterschool Program Quality and Student Outcomes: Reflections on Positive Key Findings on Learning and Development From Recent Research

In my years researching the effects of afterschool programs on children’s social and academic outcomes, I have observed the power that high quality programs can have on the learning and development of young people. This paper provides some reflections on selected research from my own study of the field in recent years, which has been deeply informed by that of many others. Since my first study of afterschool programs conducted more than 25 years ago (Vandell & Corasaniti, 1988), I am heartened by the growth in our understanding of the effects of out-of school time from a virtually unstudied area to abundant and solid evidence on the positive impacts of high quality programs. Whether they are called afterschool, expanded learning opportunities, out-of-school time, or something else, we know from research that these types of opportunities can lead to positive outcomes for children and youth, as well as families, communities, and schools (Durlak, Weissberg, & Pachan, 2011; Eccles & Gootman, 2002; Mahoney, Vandell, Simpkins, & Zarrett, 2009).


As the nomenclature in the field has evolved, so too have my own research lens and lines of inquiry. Through my investigations over the years, I have developed some beliefs about the implications of what we have learned for policy, which I share at the end of this paper. In my estimation, based on years of examination, high quality expanded learning programs are essential to the learning process because they provide young people with opportunities to relate to their world in new ways. Strong programs foster an orientation of being open to novel experiences, of being interested in others and the world, of being inquisitive and creative, and, ultimately, of becoming lifelong learners (Larson, 2000; Lerner et al., 2005; Shernoff & Vandell, 2008). As I see it, we have before us today unprecedented opportunities to ensure all expanded learning programs make a difference for children and youth (Vandell, 2012).


A Robust and Growing Research Base and 
Enhanced Measures of Effectiveness


Continued investment in research and evaluation in the expanded learning field has resulted not only in a robust research base but also in the development of reliable and valid measures of program effectiveness and impact that can be used effectively by practitioners and researchers to improve program quality (Vandell, 2011 September). Assessment tools are being created and refined by the academic and research community, as well as from within the growing local, state, and national infrastructure that promotes and supports high quality afterschool and summer programs. These instruments can be used by expanded learning programs to assess such factors as program quality and attendance; staff beliefs, attitudes, education, and training; staffing patterns, including recruitment and retention; and student performance in specific domains and skills, such as behavior and academic achievement.


The measures my colleagues and I developed for the California Afterschool Outcome Measures Project are examples of the kinds of psychometrically reliable and valid instruments available that assess student outcomes in the areas of skill development and positive behavior change (Vandell, O’Cadiz, Hall, & Karsh, 2012). The set of surveys, which can be administered online, is designed to be completed by students, program staff, and classroom teachers. Student surveys assess areas such as social competencies with peers, task persistence, work habits, and reductions in misconduct. Surveys completed by program staff and classroom teachers include measures of child behavior with other children, social skills with peers, task persistence, and work habits. With these data, programs are able to study changes in their students’ behaviors across the school year and to compare these changes to those found in other programs across the state.


In addition, students are able to use the Afterschool Outcome Measures Online Toolbox to report the quality of their experiences at the programs in three key areas—the quality of their interactions with program staff, quality of interactions with peers at the program, and their interest and engagement in program activities—again using well-established instruments with strong psychometric properties. Programs can then use these aggregated reports to assess how they are doing from the perspective of the youth who attend their program.


The Afterschool Outcome Measures Online Toolbox is now being used at more than 1,000 afterschool program sites in California, with plans to double the number of sites using the measures in the next 2 years. It will be important to see if the Afterschool Outcome Measures Online Toolbox can be used by program sites to improve student experiences (and student outcomes). 


Of course, valid and reliable measures for researchers and practitioners alike are fundamental to being able to draw conclusions about the quality and outcomes of expanded learning programs. Some of the skills and knowledge that many afterschool programs are designed to promote are, in fact, complex to assess, and research in the field is limited by the inability to use experimental design to identify causal relationships. However, the instruments, approaches, and statistical models currently available do provide us with the ability to make substantive assertions about the correlations between program quality and outcomes for students.


Program Quality and Student Outcomes—Academic, 
Social, and Behavioral


My recent research, including the Study of Promising After-School Programs (Vandell, Reisner, & Pierce, 2007), the Longitudinal Study of Program Quality (Pierce, Bolt, & Vandell, 2010), and the NICHD Study of Early Child Care and Youth Development (Li & Vandell, 2013; Auger, Pierce, & Vandell, 2013; Lee & Vandell, 2013) reinforces previous studies that the breadth, quality, intensity, and duration of expanded learning programs make a difference in both short-term and enduring effects on student academic, social, and behavioral outcomes (Mahoney, Vandell, Simpkins, & Zarrett, 2009; Vandell, 2012). Based on the evidence, following are key characteristics of high quality expanded learning programs:


• foster positive relationships between program participants and staff,

• build positive relationships among program participants,

• offer a blend of academic and developmental skill-building activities,

• promote high levels of student engagement,

• maintain an orientation toward mastery of knowledge and skills, and

• provide appropriate levels of structure as well as opportunities for autonomy and choice (Eccles & Gootman, 2002).


Other recent studies reveal that positive staff–child relations are important for both academic and socio-behavioral growth. Reading and math grades are associated with positive relationships between program staff and participants, and supportive interactions with nonparental adults are important for facilitating child adjustment. In addition, when dosage is high (that is, students attend expanded learning programs frequently and regularly), research shows that expanded learning programs can be a significant factor in fostering positive academic and social outcomes (Pierce, Bolt, & Vandell, 2010).


Other investigations (Auger, Pierce, & Vandell, 2013; Li & Vandell, 2013; Pierce, Bolt, & Vandell, 2010) that I have conducted with colleagues reinforce the finding that the availability of a diverse array of structured, age-appropriate activities is positively associated with student math grades and classroom work habits, particularly at the elementary level. As students get older and seek more autonomy in their out-of-school activities, research tells us that greater flexibility in programming becomes more important (Vandell, Reisner, & Pierce, 2007).


Some of my research sheds light on the types of activities in expanded learning programs that correlate with various student outcomes. For example, students who participate in the arts have been found to have greater self-efficacy and achievement orientation, as evidenced by their increased time doing English homework and reading for pleasure (Li & Vandell, 2013; Vandell, Pierce, & Karsh, 2011). Additionally, participation in sports seems to be associated with better work habits, self-efficacy, school attachment, and achievement orientation (Vandell, Pierce, & Karsh, 2011).


Social and behavioral outcomes. There is substantial evidence from the current body of research that expanded learning programs promote positive social and behavioral outcomes (Durlak et al., 2010). High quality expanded learning opportunities are linked to gains in social skills with peers, increased pro-social behavior, and reductions in aggression, misconduct (e.g., skipping school, getting into fights), and illegal substance use (Vandell, Reisner, & Pierce, 2007). These opportunities also demonstrate promise because they have been shown to increase student engagement, intrinsic motivation, concentrated effort, and positive states of mind (Larson, 2000; Shernoff & Vandell, 2008). These findings are significant because the social and emotional outcomes that are fostered through high quality afterschool programs lay the psychological groundwork for the kinds of cognitive processes that are required for mastery of academic content knowledge and skills to apply that knowledge.


Academic outcomes. We know from research that engagement in activities that are both fun and that require focus helps develop the competencies needed for academic learning, including concentration, intrinsic reward, and motivation (Shernoff & Vandell, 2007; 2008). For example, in the Study of Promising After-School Programs, students who regularly attended high quality programs demonstrated significant gains in standardized mathematic test scores as well as self-reported work habits (Vandell, Reisner, & Pierce, 2007). This study and other recent research provide a solid basis for three core assertions that should be used to continue to advance the field:


• Expanding learning programs show promising evidence for helping to close the achievement gap.

• High quality afterschool programs have positive long-term effects on school attendance and task persistence.

• Expanded learning opportunities have positive cumulative effects on student grades and academic work habits (Vandell, 2011 February).


Implications for Policy


One of the drivers behind my work is a strong belief that the interdependence of research, practice, and policy is key to increasing positive outcomes for children and youth. As I noted in the opening section of this paper, my research over the years has led me to form some conclusions about the research-practice-policy dynamic. Based on these, I offer the following implications of my research and that of others for practice and policy:


• Practitioners already have access to reliable and valid measures that can be used to assess program quality.

• A next step is to expand awareness in the field of these measures and to increase capacity to use these data to improve program quality and to monitor improvements in youth outcomes. 

• Practitioners can combine and compare research findings from across studies to determine the factors that fit best with their program contexts and characteristics.

• Policy makers must heed the evidence that high quality programs with sufficient dosage have positive impacts on student behavior and academic performance.

• Policy makers must set the stage for longitudinal data systems that enable the tracking of program, staff, and student indicators over time.

• Policy makers must provide sufficient resources for expanded learning programs to offer both academic activities, such as homework help, as well as enrichment activities, such as sports and arts, that ultimately help students improve academic performance (Vandell, 2010; Shernoff & Vandell, 2008).


Conclusion


Over the years, I have had the great honor to interact with a wide array of students, practitioners and educators, parents, policy makers, and other researchers in the field of expanded learning. As I reflect on the research and consider its implications for future work, I am encouraged by the growing awareness of the importance of out-of-school time as a critical educational context and by the extent and caliber of the research that is being conducted by scholars in the U.S. and elsewhere.


As we move forward together in this effort, researchers, practitioners, policy makers, and other key stakeholders, such as funders and technical assistance providers, must continue to intersect intentionally to ensure our efforts are aligned and that they inform the efforts of others. We have come a long way in having a growing body of research and evaluation evidence that quality afterschool programs work and make a positive difference. We also know a lot about improving quality. So at the local, state, and federal levels, it is time for us to find the will, energy, and resources to expand quality afterschool programs in the many schools and communities that need and want them—not in another 10 years, but now. In so doing, we will truly be able to leverage the power of expanded learning for student and community success.


References


Auger, A., Pierce, K. M., & Vandell, D. L. (2013). Participation in out-of-school settings and student academic and behavioral outcomes. Manuscript in preparation.


Durlak, J. A., Weissberg, R. P, & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294–309.


Eccles, J., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington DC: National Academy Press.


Larson, R. (2000). Toward a psychology of positive youth development. The American Psychologist, 55, 170–183.


Lee, K., & Vandell, D. L. (2013). Understanding the link between substance use and the interaction of the individual and the environment. Manuscript in preparation.


Lerner, R. M, Lerner, J. V., Almerigi, J. B., Theokas, C., Phelps, E., Gestdottir, S. . . . von Eye, A. (2005). Positive youth development, participation in community youth development programs, and community contributions of fifth-grade adolescents: Findings from the first wave of the 4-h study of positive youth development. Journal of Early Adolescence, 25, 17–71.


Li, W., & Vandell, D. L. (2013). Relating type, intensity, and quality of after-school activities to later academic and behavioral outcomes. Manuscript in preparation.


Mahoney, J. L., Vandell, D. L., Simpkins, S., & Zarrett, N. (2009). Adolescent out-of-school activities. In R. M. Lerner & L. Steinberg (Eds.), Handbook of adolescent psychology: Vol. 2. Contextual influences on adolescent development (3rd ed., pp. 228–269). New York, NY: Wiley.


National Institute of Child Health and Human Development (NICHD) Early Child Care Research Network. (2004). Are child developmental outcomes related to before- and after-school care arrangements? Results from the NICHD Study of Early Child Care. Child Development, 75, 280–295.


Pierce, K. M., Bolt, D. M., & Vandell, D. L. (2010). Specific features of after-school program quality: Associations with children’s functioning in middle childhood. American Journal of Community Psychology, 45, 381–393.


Shernoff, D. J., & Vandell, D. L. (2007). Engagement in after-school program activities: Quality of experience from the perspective of participants. Journal of Youth and Adolescence, 36, 891–903.


Shernoff, D. J., & Vandell, D. L. (2008). Youth engagement and quality of experience in afterschool programs. Afterschool Matters, Occasional Papers Series, (9), 1–11.


Vandell, D. L. (2009). California After School Outcomes Measures Project: Phase I. Submitted to the David and Lucile Packard Foundation and the Afterschool Division of the California Department of Education, University of California, Irvine.


Vandell, D. L. (2010, January). Afterschool research in 2010: Implications for policy and practice. Presentation at the 2010 National Network of Statewide Afterschool Networks Meeting, San Diego, CA.


Vandell, D. L. (2011, February). Impacts and outcomes: What we know about afterschool and expanded learning opportunities. Presentation at the 2012 Meeting of the National Network of Statewide Afterschool Networks, San Francisco, CA.


Vandell, D. L. (2011, September). The power of expanded learning opportunities today. Presentation at the 2011 National Network of Statewide Afterschool Networks Leads Meeting, Washington, DC.


Vandell, D. L. (2012, June). Expanded learning opportunities can make a difference. Presentation to the Charles Stewart Mott Foundation Board of Directors, Flint, MI.


Vandell, D. L., & Corasaniti, M. A. (1988). The relation between third graders’ after school care and social, academic, and emotional functioning. Child Development, 59, 868–875.


Vandell, D. L., O’Cadiz, P., & Hall, V. (2010, August). California Afterschool Outcome Measures Project: Phase II report. Submitted to the David and Lucile Packard Foundation and the Afterschool Division of the California Department of Education, University of California, Irvine.


Vandell, D. L., O’Cadiz, P., Hall, V., & Karsh, A. (2012, January). California Afterschool Outcome Measures Project: Report of the field test. Submitted to the David and Lucile Packard Foundation and the Afterschool Division of the California Department of Education, University of California, Irvine. Retrieved from http://afterschooloutcomes.org/sites/default/files/caomp_field_test_repo...


Vandell, D. L., Pierce, K. M., & Dadisman, K. (2005). Out-of-school settings as a developmental context for children and youth. In R. V. Kail (Ed.), Advances in child development and behavior (Vol. 33, pp. 43–77). New York, NY: Academic.


Vandell, D. L., Pierce, K. M., & Karsh, A. (2011). Study of promising after-school programs: Follow-up report to participating school districts. Irvine, CA: University of California-Irvine.


Vandell, D. L., Reisner, E. R., & Pierce, K. M. (2007). Outcomes linked to high-quality afterschool programs: Longitudinal findings from the study of promising afterschool programs. Report to the Charles Stewart Mott Foundation, Flint, MI.


Vandell, D. L., Shernoff, D. J., Pierce, K. M., Bolt, D. M., Dadisman, K., & Brown, B. B. (2005). Activities, engagement, and emotion in after-school programs (and elsewhere). In H. B. Weiss, P. M. D. Little, & S. M. Bouffard (Eds.), New directions for youth development: No. 105. Participation in youth programs: Enrollment, attendance, and engagement (pp. 121–129). San Francisco, CA: Jossey-Bass.


Vandell, D. L., Shumow, L., & Posner, J. (2005). After-school programs for low-income children: Differences in program quality. In J. L. Mahoney, R. W. Larson, & J. S. Eccles (Eds.), Organized activities as contexts of development: Extracurricular activities, after school and community programs (pp. 437–456). Mahwah, NJ: Erlbaum.