~AEA Public Library

Eval12 Session 962: Three years of data to reach intermediate outcomes and nobody will read the reports! 

10-26-2012 12:44

Abstract Data utilization is a major thrust of program evaluation efforts. In Extension, we are trained to develop logic models that lay out short-term, intermediate, and long-term outcomes. This session highlights the “The Northwest Michigan New FARM Program” that had several funding sources (federal and foundation). Multiple community partners were involved over the years, building additional stakeholders. The program amassed a large collection of data because of the multi-year evaluation plan. Vast quantities and high quality data have resulted in a series of written reports for various levels of stakeholders. However, the challenge remains to motivate intended end-users to consume the results. This program achieved intermediate outcomes that are highly sought and touted as success. This three-year, multi-method Extension program evaluation shows that often the more difficult assignment comes after results are collected, analyzed, and packaged. The hardest task is getting stakeholders to care about, understand, and use the results. Words of Relevance Data utilization is a major thrust of program evaluation efforts. In Extension, we are trained to develop elaborate logic models that lay out short-term, intermediate, and long-term outcomes based on justified inputs and outputs. Further, we spend great amounts of time to operationalize concepts that underline programming designs (e.g., content delivery effectiveness) and to find measurable indicators. This session highlights an example of a three-year effort called the “The Northwest Michigan New FARM Program.” The program had several funding sources including a large federal grant (USDA Beginning Farmer and Rancher Development Program), a generous local charities grant (Rotary Charities of Traverse City), and several other smaller contributions from local non-profits. Several funding streams mean several sets of invested stakeholders. Additionally, local media followed the program activities for years and frequently updated the public on the happenings of this leadership program of new farmers. Multiple community partners were also involved in varying degrees over the years, mostly as co-hosts and presenters of sessions, building additional interested stakeholders. The program enjoyed a fairly stable sample of program participants across time. The program amassed a large collection of data based on the multi-year evaluation plan, including: a baseline assessment upon program entry, nine process/event evaluations during the first two years to capture short-term learning changes, a mid-term evaluation, and an end-of-program evaluation that measured outcomes of a capstone international trip (agricultural tour to New Zealand) and intermediate (USDA Year 2 & 3) outcomes for participants. The entire collection of program materials and evaluation data is astounding. Vast quantities and high quality data have resulted in a series of written reports for various levels of stakeholders. However, the challenge remains to motivate intended end-users to consume the results. A broad audience can find value in the lessons learned during this three year Extension program effort because the challenges of data utilization are common to all program evaluators. The program easily demonstrated short-term outcomes, as measured by surveys post well-planned educational sessions and tours. Reaching intermediate outcomes within 3-5 years is considered benchmarked success. Program evaluation evidence clearly shows that participants in the New FARM program reached intended outcomes plus many unexpected positive changes. Yet, the issue remains to generate interest in and use of the evaluation results. Professional frustration appears when careful collection and demonstration of results of several years are articulated and the audience has limited attention to digest complexity. The voluminous qualitative narratives collected are especially difficult to summarize into sound bites. Strategies used to satisfy multiple funding streams and various partners will be discussed. The focus of the presentation is on using a real program example to discuss options for greater data utilization. Intermediate outcomes are highly sought and touted as success. What this three-year, multi-method Extension program evaluation shows is that often the more difficult assignment comes after results are collected, analyzed, and packaged into reports. The hardest task in wrapping up the program is getting stakeholders to care about, understand, and use the results.

#leadershipprogram #USDA #2012Conference #newfarmers #NIFA #EvaluationUse #ExtensionEducationEvaluation

Statistics
0 Favorited
18 Views
4 Files
0 Shares
56 Downloads

Related Entries and Links

No Related Resource entered.

Tags and Keywords

Attachment(s)
pdf file
14 photo pages with participant qoutes.pdf   3.32 MB   1 version
Uploaded - 10-26-2012
pdf file
Final Private Funder Report.pdf   487 KB   1 version
Uploaded - 10-26-2012
pdf file
Final USDA Report.pdf   554 KB   1 version
Uploaded - 10-26-2012
pptx file
New FARM presentation - AEA Oct 2012.pptx   10.01 MB   1 version
Uploaded - 10-26-2012