Our Services

American Institutes for Research (AIR) experts help maximize the impact of programs, strategies, and other interventions through innovative research methods designed to inform continuous improvement in the R&D process. Our approach to R&D is guided by the Multiphase Optimization Strategy (MOST) framework. MOST is designed to be practical, allowing developers to create potent interventions through strategic testing during the development and initial implementation process. MOST emphasizes iterative design and development. Based on MOST, our R&D approach identifies four distinct phases: development, optimization, evaluation, and scale-up.

MOST at AIR

Multiphase Optimization Strategy (MOST) at AIR

We know that many programs and strategies have multiple features and components, and end users have many different approaches to implementing them. Some interventions are adaptive, providing more support or scaffolding to those who need more intensive supports. We will help you gauge which intervention components are necessary, the ideal amount of use or exposure, and which adaptations work for whom. To learn more about how MOST has been applied in multiple fields, from engineering to prevention science, visit The Penn State Methodology Center.

Development

We understand that the development of programs, strategies, and interventions is an iterative process. The development phase often includes usability testing of materials, but also should include a feasibility study to gauge whether the developed intervention can be implemented as intended by end users in authentic settings, such as schools, classrooms, or online learning environments.

Usability: AIR conducts usability studies about user interface and user experience using such methods as observation studies, think-aloud protocols, user experience questionnaires, and mouse and eye movement tracking. In addition, we can design custom usability studies to examine the best amount, order, sequencing, labeling, and “feel” of content. The end goal is to produce interventions that are well-designed and user friendly.

Feasibility: We routinely partner with stakeholders to explore whether and how easily interventions can be implemented by end users. Small-scale feasibility studies provide invaluable information about the intervention’s potential to succeed in the authentic environment for which it was designed. The data collected from feasibility studies are useful for iterative development, and the knowledge gained can lead to a more effective intervention.

Optimization

Some interventions are rigorously tested only after years of development, and when they are, the results often are not as positive as expected. Using rigorous study designs in the early phases of development and initial implementation can provide information to maximize the impact of the intervention, all while saving cost and time. Information collected from these studies can help you “optimize” the interventions you develop or invest in.

Optimization helps tease out which components of the intervention work as intended and which do not. For example, A-B testing can give you early information about the relative effectiveness of two or more features of your intervention. Sequential Multiple Assignment Randomized Trial, or SMARTSequential multiple assignment randomized trials (SMARTs) are used to study and inform further development of adaptive interventions. Adaptive interventions are sequences of tailored decision rules that specify whether, how, or when to alter a programs, strategies, and interventions. Adaptive interventions are personalized in the sense that the intervention for any given person will change (e.g., become more or less intense) based on the person’s response to the intervention so far. studies can identify which adaptive supports are most effective. Early experimentation also can help you assess what kind of implementation models work best.

We will work with you to tailor a series of experiments to assess which components, adaptations, or implementation approaches are most promising, and which need to be improved or can be cut.

Evaluation

We will help you design and conduct a rigorous study that evaluates the effectiveness of your program.

Early Efficacy: Short, early efficacy studies can provide formative information about any intervention. Some interventions, particularly those with technology-based components that provide detailed user information or built-in assessments, may be especially good candidates for fast-turnaround or rapid-cycle experimental studies to assess early efficacy.

Effectiveness: Once your intervention is fully developed, AIR will work with you to examine its impact. We have decades of experience in designing and conducting rigorous trials that meet strict evidence standards (e.g., those used by the What Works Clearinghouse in education). We routinely partner with developers and actively participate in the process of seeking funding for effectiveness studies through federal, foundation, and other grant opportunities.

Scaling-Up

Even interventions with evidence of effectiveness may need to be redeveloped or adapted over time for the purpose of scaling up or to fit with changes in practices or context. We have the content knowledge, applied experience, and methodological expertise to help reconceptualize interventions and approaches to implementation in order to expand reach.

Measurement

Measurement is key to every phase of R&D—without valid or reliable measures, you can’t know if your intervention is having the intended effect. AIR will work with you to identify or develop the appropriate measures for your particular study. This is an important challenge for rapid-cycle, iterative studies that require more frequent measures of proximal outcomes. Our approach identifies the outcome measures best aligned with the goals of your intervention, with an emphasis on novel, unobtrusive measurement opportunities that make use of readily accessible user dataFor example, AIR can apply machine learning techniques (data mining and learning analytics) to educational technology-usage data to measure engagement and progress through learning trajectories. .

For educational interventions, AIR has extensive expertise in developing and using measures of student achievement, classroom practice, school climate, and social-emotional learning. AIR experts also create and capture relevant measures for studies in the areas of health, workforce and labor, and prevention.