Evaluation 2020

Session 2078. Illuminating Lessons Learned in a Complex Multi-Site Program Evaluation Using the RE-AIM Framework 

10-22-2020 01:42

The RE-AIM framework, which is increasingly being used in evaluating community-based prevention programs, emphasizes five dimensions: Reach, Effectiveness, Adoption, Implementation, and Maintenance. We used secondary datasets to evaluate the public health impact of the Community Prevention and Wellness Initiative (CPWI) using the RE-AIM framework. CPWI is a complex, multi-site, multi-level adolescent substance use prevention initiative in Washington State. This poster will describe how we generated the evaluation questions for each RE-AIM dimension, selected variables from existing datasets including administrative data, conducted the analysis, calculated the RE-AIM score, and set up criteria for low/medium/high score threshold for each dimension. We will also highlight successes, challenges, and lessons learned in this project. This demonstration will be useful for evaluators leading complex multi-site evaluations who are interested in incorporating the RE-AIM framework in their evaluation practice.

Relevance

The RE-AIM framework is an ideal option to evaluate multi-site, multi-level projects because the framework not only focuses on program effectiveness, but also on the degree to which a program can be adopted and disseminated widely, have consistent implementation at a reasonable cost across settings to reach a large number of the target population, and have the ability to be maintained or sustained over time. Thus, the RE-AIM framework will allow evaluators to comprehensively assess the who, what, when, where, how, and why of an initiative (Glasgow, Vogt, & Boles, 1999; Harden et al., 2018). Each dimension of the RE-AIM framework guides program planning and evaluation efforts. Though it is ideal to employ the full RE-AIM model to assess the overall public health impact of an intervention, it allows for the measurement of specific dimensions based on evaluation needs (Harden et al., 2018).

There is a growing interest among evaluators to use the RE-AIM framework in community-based settings (Glasgow et al., 2019). We will use our evaluation of the CPWI initiative to demonstrate how the RE-AIM framework can be used for evaluating complex, large-scale initiatives. CPWI is a strategic, data-informed, community coalition model aimed at bringing together key local stakeholders to provide the needed infrastructure and support to successfully coordinate, assess, plan, implement, and evaluate youth substance use prevention services needed in their community. The primary long-term outcome of interest for CPWI is reducing youth behavioral problems such as substance use and related risk factors. Each CPWI community coalition is required to implement an environmental strategy, a public awareness campaign, and direct services targeting youth substance use. Currently, there are 81 high-need communities in various stages of CPWI implementation in all 39 counties and across all 9 educational service districts in Washington State.

A majority of RE-AIM evaluations focus on individual dimensions (Harden et al., 2018). Our work addresses this gap, contributing to evaluation practice by providing a primer of how the RE-AIM framework, as a whole, can be applied to evaluating large-scale initiatives. We will demonstrate how we operationalized each of the RE-AIM dimensions, developed evaluation questions, leveraged existing administrative data as well as statewide surveillance data on youth health behaviors, developed RE-AIM score threshold and analyzed data. Evaluators might be especially interested in learning more about how we used propensity score weighted multilevel modeling to assess the ā€œEā€ (i.e., Effectiveness) of CPWI. We will also present the results of our evaluation to demonstrate how to visually represent the results of the RE-AIM dimension. In order to provide recommendations for applying the RE-AIM framework for large-scale initiatives, we will also share perspectives and lessons learned from employing the RE-AIM framework. Even though our focus is on community-based programs, the processes, perspectives, and lessons learned can be applied to evaluating programs in other settings.

 

 

 

Statistics
0 Favorited
7 Views
2 Files
0 Shares
3 Downloads

Related Entries and Links

No Related Resource entered.

Tags and Keywords

ā€‹
Attachment(s)
pdf file
Session ID 2078 Shrestha et al. RE-AIM poster   783 KB   1 version
Uploaded - 10-22-2020
pdf file
Session ID 2078 Shrestha et al. RE-AIM handout   460 KB   1 version
Uploaded - 10-22-2020