Rapid-Cycle Projects
A number of AIR projects demonstrate our flexibility and creativity in designing tailored studies to address critical questions related development, optimization, and effectiveness. Along all phases, we employ rigorous research methods to applied problems of practice, to generate high quality evidence-based information that guides decision-making in a continuous improvement process.
Impact Evaluation of Parent Messaging Strategies on Student Attendance
AIR and our partners from the University of Chicago, the North Carolina State University, and 2M are evaluating the use of text messaging for informing parents about the importance of regular student attendance and increasing parent engagement.
The goal of the parent messaging strategies is to reduce chronic absenteeism among students in elementary schools. The parent text messaging intervention is designed to be adaptiveWhat is an Adaptive Intervention?
Adaptive interventions are sequences of tailored decision rules that specify whether, how, or when to change the intervention based on how participants respond along the way. Adaptive interventions are personalized in the sense that the intervention for any given person will change (or not) based on their progression, determined at specified time points, by the person’s response to the intervention so far in the sense that parents whose children miss more school during the school year will receive more intensive messaging strategies.
This cutting-edge study, funded by the U.S. Department of Education, is evaluating the effectiveness of the adaptive parent messaging strategies using a sequential multiple assignment randomized trial (SMART) design. The study will evaluate the overall impact of the two-stage intervention on student attendance rates and student achievement. The SMART design also will allow us to examine the relative impact of the different intervention sequences, for different types of families and children. In total, approximately 60 elementary schools and 30,000 students and their parents will participate.
Efficacy Study of Online Credit Recovery in Algebra I
AIR conducted a short-duration randomized controlled trial in the Chicago Public Schools to test the impact of using an online course for credit recovery. The study was designed to address a pressing problem in U.S. education, regarding the effectiveness of online courses for helping students who fail courses get back on track for graduation. Online courses are in wide use for this purpose because they are convenient and flexible, but evidence is sorely lacking.
Our study used a simple lottery to randomly assign more than 1,200 students who had failed Algebra I in ninth grade to retake the course as either an online course or a face-to-face class. The courses were offered during the summer, in three- to four-week sessions—typical for credit recovery. Within a span of six to eight weeks, we randomly assigned eligible students to one of the two types of credit recovery, documented how the courses were implemented, collected data for outcome measures, and conducted preliminary analyses.
The project was funded with a five-year grant that allowed us to continue to follow the students all the way through high school, but at the heart of the study was the short-duration trial that compared the relative effects of online versus face-to-face credit recovery on students’ engagement and attitudes, algebra learning, and pass rates. These types of quick but rigorous studies are feasible, inexpensive, and yield high-quality, dependable information.
Evaluation of Playground Physics
AIR worked closely with the New York Hall of Science, as a research and development partner, to support and inform the development of Playground Physics, a technology-based application and aligned curriculum materials. The program is designed to support middle school students’ learning of the key physics concepts of motion, force, and energy. The program is meant to be implemented during the physics units of middle school science courses, typically about two to four weeks in duration, thus readily lending itself to rapid-cycle studies.
Our study was tailored to meet the needs of the developers at multiple stages of development, refinement, and evaluation. The Implementation Study, conducted during the 2014–15 school year, focused on feasibility and usability. We examined the experiences of 22 teachers and their middle school students with the Playground Physics program with surveys, observations, and interviews. The information we gathered during this stage informed the iterative improvement of the program throughout the year.
The Impact Study, conducted in 2015–16, used a lottery to randomly assign 60 middle school science teachers either to a treatment group or to a wait-list control group. Teachers in the treatment group received Playground Physics professional development and were asked to implement the program as part of their physics instruction. Teachers in the control condition were asked to provide instruction on physics concepts as they had in previous years (business as usual). Using this design, we examined the impact of Playground Physics on student engagement and attitudes toward science education (measured through pre-post student surveys) and knowledge of standards-aligned science content (measured through pre-post knowledge assessments). We administered pre-tests in the early fall in all classes and post-tests on a rolling basis, as each participating class completed the roughly two to four weeks of physics instruction. We observed variation in how teachers blended the Playground Physics curriculum with existing resources, and, overall, we found that Playground Physics improved students’ physics knowledge although not their self-reported engagement in classroom activities or attitudes toward science.
Pathways to Success
Pathways to Success is a 12-session social psychological intervention that helps students identify images of who they may become in both the near and distant future, connect those future images to current strategies in school, and learn to reframe difficulty in school as meaning “this is important” rather than “this is impossible.” In a prior efficacy trial conducted in Detroit Public Schools, Pathways to Success had a positive impact on student attendance, grades, and effort in school. But although the intervention was effective, it was not scalable.
Further development was designed to improve scalability. We focused on transitioning the program from an external trainer model (delivered by individuals outside of the school, trained and employed by the developer) to a teacher-led, and teacher-trained model in Chicago Public Schools.
In partnership with the University of Southern California, AIR used an iterative design to obtain feedback on teacher implementation and student outcomes across rapid six-week implementation cycles. Working closely over 30 teachers in 10 schools, and over 1,000 students across three years, we conducted video-observations of implementation of all sessions delivered by teachers and in-depth interviews with teachers to assess user experience, and generated quick-turnaround feedback and actionable recommendations to improve components of the intervention. We also assessed pre-post student changes in key psychological processes, and incorporated a quasi-experimental assessment of student academic outcomes for each iteration of development and implementation. This formative rapid-cycle iterative evaluation guided major refinements to all intervention resources (manual, video library of implementation, materials for activities), which resulted in improved usability, feasibility, and scalability.